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Preface 


Despite being nearly 500 years old, with work dating back to Cardano, 
Euler, Gauss, Cauchy, Riemann, and many others, the subject of complex 
analysis is still today a vital and active part of the mathematical sciences. In 
addition to all the exciting theoretical work being done today, there are 
important applications to physics, engineering, cosmology, and other 
aspects of technology. Many of the world's most distinguished and 
accomplished mathematicians conduct research in complex analysis. 
Several recent Fields Medalists study complex analysis. 

Although a venerable subject, complex analysis continues to grow and 
prosper. New directions of development in the subject include dynamical 
systems, quasiconformal mappings, harmonic measure, automorphism 
groups, and the list can go on at some length. One of the sources of strength 
for the subject is its interaction with diverse parts of mathematics, including 
differential geometry, partial differential equations, functional analysis, 
algebra, combinatorics, and many other aspects of the subject. 

This Handbook of Complex Analysis presents contributed chapters by 
several distinguished mathematicians, including a new generation of 
researchers. More than a compilation of recent results, this book offers a 
stepping stone for students to gain entry into the professional life of 
complex analysis. The essays presented here are all accessible to graduate 
students but will also be of considerable interest to the seasoned 
mathematician. Classes and seminars, of course, play a role in the 
maturation process that we are describing. But more is needed for the 
unilateral study. This handbook will play such a role. 


As noted, this book will serve as a reference and a source of inspiration 
for mature mathematicians—both specialists in complex analysis and others 
who want to become acquainted with current modes of thought and 
investigation. And it will help the neophyte to become inured in the subject 
matter. 

The chapters in this volume are authored by leading experts in the 
subject area, also gifted expositors. They are carefully crafted presentations 
of diverse aspects of the field, formulated for a broad and diverse audience. 
The editor intends this volume to be a touchstone for current ideas in the 
broadly construed subject area of complex analysis. It should enrich the 
literature and point to some new directions. The point here is not to present 
an epitaph for complex analysis but rather to provide an entree to a whole 
new life. We anticipate that the reader of this volume will be eager to 
explore other parts of complex analysis literature and to begin to play an 
active role in complex analysis research life. 


Editor 


Steven G. Krantz is a professor of mathematics at Washington University in 
St. Louis. He has previously taught at UCLA, Princeton University, and 
Pennsylvania State University. He has written more than 130 books and 
more than 250 scholarly papers and is the founding editor of the Journal of 
Geometric Analysis. An AMS Fellow, Dr. Krantz has been a recipient of the 
Chauvenet Prize, Beckenbach Book Award, and Kemper Prize. He received 
a PhD from Princeton University. 
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Se 
1.1 Mathematical DNA 


For reasons that have always been mysterious to the first author, he has often found himself thinking about the 
Dirichlet problem in the plane and has fought urges to look for explicit formulas for the Poisson kernels associated 
to various kinds of multiply connected domains, especially quadrature domains. He has obsessed about solutions to 
the Dirichlet problem with rational boundary data (see [4]), and he cannot stop thinking about the Khavinson- 
Shapiro conjecture about the same problem with polynomial data. His publication list is interspersed with papers 
where he has given in and found formulas for the Poisson kernel in terms of the Szeg6 kernel and in terms of 
Ahlfors maps. (See [2, 3] for an expository treatment of some of these results.) After thinking about the Dirichlet 
problem his entire adult life, he can solve it a different way every day of the week. He recently looked up his 
mathematical lineage at the Math Genealogy Project and found a possible explanation for his obsession. He is a 
direct mathematical descendent of both Poisson and Dirichlet. These problems are in his blood! 

The authors worked together on a summer research project at Purdue University in 2018 to find a particularly 
elegant and simple way to approach these problems. We want to demonstrate here how Poisson and Dirichlet might 
solve their famous problems today if they had lived another 200 years and developed a major lazy streak. We 
assume that our reader has seen a traditional approach to this subject in a course on complex analysis and so will 
appreciate the novelty and smooth sailing of the line of reasoning here, but just in case the reader hasn’t, we have 
tried to present the material in a way that can be understood assuming only a background in basic analysis. 

We would like to thank Harold Boas for reading an early draft of this work and making many valuable 
suggestions for improvement. Harold knows a lot about the family business because he, like the first author, was a 
student of Norberto Kerzman at MIT in the late 1970s. 


1.2} Harmonic Functions 


We define a harmonic function on a domain Q in the complex plane to be a continuous complex valued function 
u(z) on the domain that satisfies the averaging property on , meaning that 


ar 
u(a) = a! u(a + re”) dO 


whenever D,(a), the disc of radius r about a, is compactly contained in Q. It is well known that this definition of 
harmonic function is equivalent to all the other standard definitions (see, for example, Rudin [7, Chap. 11]), and we 
will demonstrate this in very short order. Since we will explore other definitions of harmonic, we will emphasize 
that we are currently thinking of harmonic functions as being defined in terms of an averaging property by calling 
them harmonic-ave functions. 

We first note that analytic polynomials are harmonic-ave. Indeed, if 
P(z) = Gn2" + Qn12"1+...+€12+ a9, the averaging property is clear on discs centered at the origin 
because the constant function a) obviously satisfies the averaging property at the origin and so does z” for n > 1 
because 


20 20 20 
/ (re®)" do =r” i cos(n6) dO + ir” i sin(n@) d0 = 0 = 0”. 
0 0 0 
To see that the averaging property holds at a point a € C, write 
P(z) = P((z-—a)+a) 


and expand to get a polynomial in (z— a). Now the argument we used at the origin can be applied to discs 
centered at a. 

It follows that the averaging property also holds for conjugates of polynomials in z, so polynomials in Z are 
harmonic-ave. Note also that complex valued functions are harmonic-ave if and only if their real and imaginary 
parts are both harmonic-ave. 

The Dirichlet problem on the unit disc is: given a continuous real valued function @ on the unit circle, find a 
real valued function u that is continuous on the closure of the unit disc with boundary values given by @ such that 
u is harmonic on D;(0). We will find a solution to this problem that is harmonic in the averaging sense in the next 
section. To do so, we will need to know the elementary fact that a continuous real valued function on the unit circle 
can be uniformly approximated on the unit circle by a real polynomial p(z, y). We will now demonstrate this little 
fact, assuming the Weierstrass theorem about the density of real polynomials of one variable among continuous 
functions on closed subintervals of the real line. 

Suppose we are given a continuous real valued function @ on the unit circle. We wish to find a real polynomial 


p(z, y) that is uniformly close to @ on the unit circle. Note that we may assume that ¢(-+1) = 0 because we may 


subtract a polynomial function of the form az + b to make @ zero at +1. Define two continuous functions on 
[—1, 1] via 


hiop(@) = o(e +ivI— 2?) 


and 


Apot (x) = o(@ —iv1 — 2). 


We can uniformly approximate h,,, and 4,,, on [—1, 1] by real polynomials pjop(z) and ppot (x). Next, let x<(y) be 


top 
a continuous function that is equal to one for y > €, equal to zero for y < —€ and follows the line connecting 


(—€,0) to (€,1) for —€ < y < €. Let pe(y) be a polynomial in y that is uniformly close to y, on [—1, 1]. Now, 


because @ vanishes at +1, the polynomial 


Ptop(@)Pe(y) + Poot (x)Pe(—y) 


approximates @ on the unit circle, and the approximation can be improved uniformly by shrinking € and improving 
the approximations of the other functions involved. 

We are now in position to solve the Dirichlet problem on the disc, but before we begin in earnest, this is a good 
place to emphasize that Green's theorem for the unit disc depends on nothing more than the fundamental theorem 
of calculus from freshman calculus. Indeed, let P(x, y) be a C!-smooth function. Let Crop denote the top half of 
the unit circle parameterized in the clockwise sense by z(z) =2+tytop(x), —l1<a<1, where 


Ytop(x) = V1— 2”, and let G,,, denote the bottom half of the unit circle parameterized in the counterclockwise 


sense by z(z) = 2x+ typor(z), —l1 <a <1, where ypo(x) = —/1—2?. Note that the unit circle C;(0) 
parametrized in the counterclockwise sense is given by C,,, followed by —C'op. Drum roll... 


Ytop(x) OP 
i} ar (x, g 
Ybot (x) y 


on i 7 / sre tes I [P(#, Yoor(#)) — P(t, Ytop(2))] dar & = — / | ( 


af ot 


The other half of Green's formula follows by repeating the argument using the words left and right in place of top 
and bottom. This argument on the disc can be easily generalized to demonstrate Green's theorem on any region that 
can be cut up into regions that have a top boundary curve and a bottom curve and a left curve and a right curve. An 
annulus centered at the origin cut into four regions by the two coordinate axes is such a domain. We will need 
Green's theorem later in the paper when we study analytic functions from a philosophical point of view inspired by 
our observations about harmonic functions and the Dirichlet problem. 


1.3 Solution of the Dirichlet Problem on the Unit Disc 


The unit disc has the special feature that given polynomial data @, it is straightforward to write down a polynomial 
solution to the Dirichlet problem with boundary values given by @. Indeed, a polynomial p(z, y) in the real 
variables x and y can be converted to a polynomial in z and Z by replacing x by (z+ Z)/2 and y by (z — Z)/(2i) 
and expanding. It is now an easy matter to extend the individual terms in the sum to harmonic-ave functions on the 
disc by noting that 


gigs 


is equal to one on the unit circle ifm = m, equal to z”~™ on the circle if n > m, and equal to 7™~” on the circle if 
m > n, each of which is harmonic-ave inside the unit circle. 

Now, given a continuous real valued function @ on the unit circle, there is a sequence of real valued 
polynomials p,(x,y) that converges uniformly to @ on the unit circle. Let u,, be the polynomial harmonic-ave 
extension of p, to the disc described in the paragraph above. Note that u, can be expressed as a constant plus a 
polynomial in z that vanishes at the origin plus a polynomial in Z that also vanishes at the origin. We now claim 
that the functions u, converge uniformly on the closed disc to a solution of the Dirichlet problem. To see this, we 


must first show that real valued harmonic-ave functions u on the disc that extend continuously to the closure satisfy 
the maximum principle in the form 


max{u(z) : |z| < 1} = max{u(e”) :0 <0 < 2r}. 


Indeed, if the maximum value of such a function u occurs at a point Zp inside the unit circle, we can express the 
value of u at Z) as an average of u over a small circle centered at z. We can let the radius of that circle increase 
until the circle touches the unit circle at a single point. The averaging property holds on the limiting circle because 
of uniform continuity. Let M denote the maximum value u(zo). Now, in order for the average of a continuous 
function that is less than or equal to M over that circle to be equal to M, it must be that u is equal to M on the 
whole circle. Hence the value of u at the point where the inner circle touches the unit disc must also be M. This 
proves the maximum principle inequality for real harmonic-ave functions. The minimum principle follows by 
applying the maximum principle to —u. 


Because the p, converge to @ uniformly on the unit circle, the sequence {p,,} is uniformly Cauchy on the unit 


circle, i.e., given € > 0, there is an N such that |p,, — pm| < € on the unit circle when n and m are greater than N. 
The maximum and minimum principle inequalities applied to the imaginary parts of u, (which are zero on the unit 


circle) allow us to conclude that the functions u, are real valued. Furthermore, the maximum and minimum 


principles applied to un — um show that the uniformly Cauchy estimates for the sequence {p,,} on the unit circle 
extend to hold for the sequence {u,,} on the whole closed unit disc, showing that {u,,} is uniformly Cauchy on the 
closed unit disc. Hence, since the u, are continuous, they converge uniformly on the closed unit disc to a 


continuous function u that is equal to @ on the unit circle. Finally, it is clear that u is harmonic-ave on the inside of 
the unit circle because the averaging property is preserved under uniform limits. We have solved the Dirichlet 
problem in a purely existential manner without ever differentiating a function! We now turn to Poisson's problem 
of finding a formula for our solution. 


| st 
1.4 Poisson's Formula 


Notice that the set of harmonic-ave functions 


is orthonormal under the inner product 


Define Ky(z, w) via 
N N 
Ky(z,w):=1+ SS 2p + S> Zw", 


and observe that if u(z) = 2”, then 


(1.1) 


for z € D,(0) if N > n because of the orthonormality of the terms in the sum. The same is true if u(z) = 1 or 
u(z) = Z". We conclude that if u is the solution of the Dirichlet problem for polynomial data p of degree n as 
constructed in the previous section, then formula (1.1) holds for u(z) for z € D(0) when N > n. 

Using the famous geometric series formula, 


we see that 
N-1 N-1 
Ky(z,w) =14 (zw) zw" + (zw) S$) z"w" 
n=0 n=0 


converges uniformly in w = e when z € D,(0) to 


K(z,w) :=1+ {+ 


and the error &y = |Ky — K| is controlled via 


Q|z\Nt4 


Ey (z,w) < ae 


when z € D,(0) and |w| = 1. Hence, it follows by taking uniform limits that formula (1.1) holds for z € Dj(0) 
with the Nremoved for the polynomials u,, that we constructed from polynomial boundary data p,. We can now let 


the polynomials p,, tend uniformly to @ and use the fact proved above that the corresponding solutions u,, to the 
Dirichlet problem with boundary data p, converge uniformly to a solution u of the Dirichlet problem to obtain 
Poisson's famous formula for the solution to the Dirichlet problem, 


(1.2) 


This formula reveals that the solution u can be written 


u(z) = ao + A(z) + A(z) 


where ap is the (real valued) average of @ on the unit circle and A is an analytic function on D,(0) that vanishes at 
the origin given via 


ZE 


Qn —i0 : (w) 
ee | 10 _ 2 
A(z) = 3, i (se d(e") dd = 3% a w(w — 2) dw, 


where C, denotes the unit circle parameterized in the standard sense using w = e”” and dw = ie“ d6. It is now a 


rather easy exercise to take limits of complex difference quotients to see that complex derivatives in z can be taken 
under the integral sign in the definition of h. Hence, h(z) is infinitely complex differentiable. It follows from the 
Cauchy-Riemann equations that u is a C'~-smooth real valued function on D,(0) in x and y that satisfies the 
Laplace equation there and solves the Dirichlet problem. Furthermore, u is the real part of an infinitely complex 
differentiable function H = ag + h/2 on D,(0). 

The maximum principle yields that u is the unique solution to the problem in the realm of harmonic functions 
understood in the sense of averaging. We will see in the next section that it is also the unique solution among 
harmonic functions defined in the traditional sense of satisfying the Laplace equation. 

We remark here that rather simple algebra reveals the well-known formulas for the Poisson kernel, 


K(z,w) = Re {#22 = Re “42 
ZW WZ 


and 


It is a routine matter to extend the above line of reasoning to any disc (either by repeating the argument or 
making a complex linear change of variables Az + B). Since a complex valued function is harmonic-ave if and 
only if its real and imaginary parts are harmonic-ave, it follows from our work that a complex valued harmonic-ave 
function is given locally by g + G where g and G are infinitely complex differentiable functions. 

At this point, it would be tempting to experiment with thinking of analytic functions as being harmonic-ave 
functions that do not involve the antianalytic G parts. The formula for h above reveals that the analytic g part is 
locally a uniform limit of analytic polynomials. Since {1, z” : n = 1, 2,3,...} are orthonormal on the unit circle, 
we could let 


N 
ky(z,w) =1+ S> 2p" 
n=1 
and use the same line of reasoning that we used earlier in this section to conclude that 
20 ; : 
fe) = 4 [kv ee) se) a0 
0 


(1.3) 


if f(z) is equal to 1 or z” with n < N. The geometric series estimate we used above shows that ky(z,w) 
converges uniformly in won the unit circle for fixed z in D (0) to 


k(z,w) := 


1-zw* 


Hence, we can let N —> oo in (1.3) to see that 


(1.4) 


when fis an analytic polynomial. Finally, if fis a uniform limit of analytic polynomials on an open set containing 
the closed unit disc, we may conclude that f satisfies identity (1.4), too. The identity can easily be seen to be the 
classical Cauchy integral formula on the unit disc, and from this point, the theory of analytic functions would gush 
forth. In particular, analytic functions would be seen to be infinitely complex differentiable and given locally by 
convergent power series. We will explore this idea and various other alternate ways of thinking about analytic 
functions after we verify that our definition of harmonic functions via an averaging property gives rise to the same 
set of functions as any of the more standard definitions. 


———————————————— 
1.5 Traditional Definitions of Harmonic Functions 


Some complex analysis books define harmonic functions to be twice continuously differentiable functions that 
satisfy the Laplace equation. With this definition, one can use Green's identities on an annulus (which we pointed 
out in section 1 to be quite elementary) to show that such harmonic functions satisfy the averaging property. 
Hence, this class of functions could be seen to be the same as harmonic-ave functions. We won’t pursue this idea 
here because we can easily prove something stronger with less effort. 

It is most gratifying to define harmonic functions to be merely continuous functions u whose first partial 
derivatives exist and whose second partial derivatives 0?u/Ox? and 0?u/dy” exist and satisfy the Laplace 
equation. Call such functions harmonic-pde. We will now adapt a classic argument to show that the class of 
harmonic-pde functions agrees with our class of continuous functions that satisfy the averaging property on an 
open set. Indeed, if u is a harmonic-pde function defined on an open set containing the closed unit disc, then u 
minus the harmonic-ave function U that we constructed solving the Dirichlet problem on the unit disc with the 
same boundary values as u on the unit circle, if not the zero function, would have either a positive maximum or a 
negative minimum. Suppose it has a positive maximum M > 0 at a point z) in D,(0). Choose €withO<e< M. 


Now 
v(z) = u(z) — U(z) + €|z|? 


is equal to € on the unit circle and attains a positive value at Zz that is larger than €. Hence, v attains a positive 
maximum at some point wo in D;(0). The Laplacian of v at wo is equal to 4€, which is strictly positive. However, 


the one variable second derivative test from freshman calculus applied to v in the x-direction yields that 0?v/0x? 


must be less than or equal to zero. (If it were positive, v could not have a local maximum at Wo.) Similarly, the 


second derivative test in the y-direction yields that 0?v/Oy? must be less than or equal to zero. We conclude that 
the Laplacian of v at wy must be less than or equal to zero, which is a contradiction because the Laplacian is 


strictly positive on D,(0). This shows that wu — U cannot have a positive value. If we replace u — U by U — u, the 
same reasoning shows that U — u cannot have a positive value. Hence U — u=0 on the unit disc, and we have 
shown that u, like U, is harmonic in the sense of being averaging. Finally, because the operations of translating and 
scaling preserve harmonic functions in both senses of the word, we can translate and scale any disc to the unit disc 
to be able to deduce the equivalence of the two definitions of harmonic on any open set. We now stop hyphenating 
harmonic and turn to hyphenating analytic. 


————SSSS— See 
1.6 Analytic Functions 


When the 200+ year old Poisson and Dirichlet took a break from harmonic functions, they might have turned their 
attention to similar considerations applied to analytic functions, which satisfy both the averaging property on discs 
and Cauchy's theorem for circles. 

In order to understand the rest of this section, the reader will need to know (or look up) the definition of the 


complex contour integral / f dz along a curve y, the basic estimate 


5 
[te 


and the fundamental theorem of calculus i f' dz = f(b) — f(a) for complex contour integrals, assuming that fis 


:Z€E v} - Length (7), 


< sup Ko 


continuously complex differentiable and that y starts at a and ends at b. We will also use the differential operators 


Oe [aes = 8 i= 2) o.4 Geo 
pai(Z if) and £=4( £422). 


These two very important operators can be “discovered” by writing dz = dx +idy and dz = dz —idy and 
manipulating 


df = Fh da + Zf dy 
to appear in the form 
df = Ldz+ Laz. 


The condition a 
ae 


If h(z) is a complex differentiable function, then gh = 0 and gh = h'(z). Furthermore, oh = 0 and oh = hi(z). 
We will denote the boundary circle of the disc D,(a) parameterized in the counterclockwise dence by C,(a). 
Writing out the real and imaginary parts of the contour integral of a complex function faround C,(a) and applying 


Green's theorem for a disc to the real and imaginary parts yields, what we like to call, the complex Green's theorem 


for a disc, 
ha fdz=2i | less cf dA, 


where dA denotes the element of area dx \ dy. (Using the real Green's theorem to prove the complex Green's 
theorem is quick and easy, but a more civilized way to deduce the theorem would be to note that 


= 0 is equivalent to writing the Cauchy-Riemann equations for the real and imaginary parts of f 


d(f dz) = 44 dzndz+ 44 dz dz=0+4 4 (2idz A dy) 


and to apply Stokes’ theorem.) 

Define a continuous complex valued function fto be analytic-circ on a domain OQ if it is harmonic on ( (and so 
satisfies the averaging property) and, for each closed subdisc of 0, the complex contour integral of f over the 
boundary circle is zero. 

Complex polynomials are easily seen to be analytic-circ because the monomials are complex derivatives of 
monomials one degree higher and the fundamental theorem of calculus for complex contour integrals reveals that 
the integrals around closed curves are zero. It now follows that functions that are the uniform limit of complex 
polynomials (in z) on each closed subdisc of 0 are analytic. We will now show that functions that are analytic-circ 
must be the uniform limit of complex polynomials on each closed subdisc of 0. 

Suppose fis analytic-circ on 0. Let D,(zo) be a disc that is compactly contained in Q and let C,.(zo) denote 
the boundary circle parameterized in the counterclockwise sense. The complex Green's theorem for a disc yields 


that 
0 
o- | faz=2i ff OF aa. 
C,(z0) D,(z) 9% 


where dA denotes the element of area dx / dy. Since this integral is zero for every disc compactly contained in Q, 
it follows that Of /OZ = 0 on Q, i.e., that the real and imaginary parts of f satisfy the Cauchy-Riemann equations. 
Since harmonic functions are infinitely differentiable, the textbook proof that fis complex differentiable can now 
be applied. 

Since being analytic-circ is a local property and is invariant under changes of variables of the form Az + B, we 
may restrict our attention to a function fthat is analytic-circ on a neighborhood of the closed unit disc. We can gain 


more insight into the implications of the definition by noting that such a harmonic function is given by g + G 
where g and G are infinitely complex differentiable. It follows from our complex Green's calculation above that 
Of /OZ = G' = 0 on Q, and that fis therefore equal to the analytic part g plus a constant. From this point, it 
follows that fis given by a Cauchy integral formula, and we merge into the fast lane of the classical theory of 
analytic functions. 

Of course, the traditional way to define analytic functions is as complex differentiable functions on open sets. 
Call such functions analytic-diff. We will now show that the traditional definition leads to the same class of 
functions that we have defined as being analytic-circ. 

Suppose that f is an analytic-diff function defined on an open set containing the closed unit square 
SF := [0,1] x [0,1]. We will show that the complex integral of faround the counterclockwise perimeter curve o of 
the square must be zero. The well-known argument will be a beautiful bisection method tracing back to Goursat. 

It follows from our assumption that f can be locally well approximated by a complex linear function in the 
following sense. Suppose a is a point in Y, and let € > 0 be given. Since f'(a) exists, there is ad > 0 such that 


f(2=-fl@) _ f'(a) + E,(z) 


Zz—-Qa 


where |E,(z)| < € when |z— a| < 6, z #.a. Define E,(a) to be zero to make E, continuous at a and so as to be 
able to assert that 


F(z) = F(a) + F'(a)(z — a) + Ealz)(z — a) 
on the whole open set where fis defined and E, is a continuous function in z on that set. Furthermore, |E4(z)| < € 
on D(a). The complex integral of the polynomial f(a) + f’(a)(z— a) around any square is zero because first- 
degree polynomials are derivatives of second-degree polynomials. If o, is the counterclockwise boundary curve of 
a small square ., of side A contained in D;(a) that contains the point a, it follows that 


[ f(z) dz 


< €(V/2h) (4h). 


jf. E,(z)(z — a) dz 


Hence, 


< (4/2 )e Area (.%,). 


[ ta: 


(1.5) 


We will now follow a version of Goursat's famous argument to explain how this could be made too small if / fdz 
oO 
were not zero. 


Indeed, suppose that I := / f dz is not equal to zero. Note that Jis equal to the sum of the integrals around 


the four counterclockwise squares obtained by cutting the big square into four equal squares of side 1/2 since the 
integrals along the common edges cancel. For these four integrals to add up to the non-zero value J, the modulus of 
at least one of them must be greater than or equal to |I|/4. Name such a square .~, and its counterclockwise 
boundary curve o,. Note that 
/ fdz 
O71 


We may now dice up .~ into four equal subsquares and repeat the argument to obtain a square .“2 with 
boundary curve o> such that the modulus of the integral of faround a, is greater than or equal to |I| times the area 


2 


I|- Area (.“%). 


of “2. Continuing in this manner, we obtain a nested sequence of closed squares 151, 4 with boundary curves 
0, the diameters of which tend to zero as n — oo, such that 


/ fdz 


z 


I|- Area (A). 


(1.6) 


There is a unique point a that belongs to all the squares. Now, given an € less than |Z | / (4V/ 2), the squares that 
eventually fall in the disc Ds(a) that we specified above satisfy both area inequalities (1.5) and (1.6), which are 
incompatible. This contradiction shows that J must be zero! 

Since any square can be mapped to the unit square via mapping of the form Az-+ B, it follows from a simple 
change of variables that the integral of an analytic-diff function around any square must be zero. Furthermore, any 
rectangle can be approximated by a rectangle subdivided into a union of n x m squares. It follows that the integral 
of an analytic-diff function around any rectangle must be zero. 

From this point, there are several standard arguments to prove the Cauchy integral formula on a disc for such 
functions (see Ahlfors [1, p. 109] or Stein [9, p. 37]). It then follows from the Cauchy integral formula that such a 
function would be locally the uniform limit of analytic polynomials, and so the function would be analytic-circ. 
However, we can simplify these standard arguments by using some of the power of our work on harmonic 
functions in the previous sections. 

Suppose that fis analytic-diff on an open disc. Since fis complex differentiable, it is continuous. Define F(z) 
at a point z in the disc by the integral of f along a horizontal “zig” from the center followed by a vertical “zag” 
connecting to the point z. The fundamental theorem of calculus reveals that 


HF (a + iy) =if(x + ity). 


Since the integral of f around rectangles is zero, we could also define F via an integral along a vertical zag 
followed by a horizontal zig. Using this definition, the fundamental theorem of calculus shows that 


ZF(z + iy) = f(x + ty). 


Hence, F is a continuously differentiable function whose real and imaginary parts satisfy the Cauchy-Riemann 
equations, and furthermore, is a complex differentiable function such that F’ = f on the disc. Repeat this 


construction to get a twice continuously complex differentiable function G such that G” = f. 

Now, since G is twice continuously complex differentiable, it is easy to use the Cauchy-Riemann equations to 
show that the real and imaginary parts of G are harmonic functions. Indeed, if G(a + iy) = u(z, y) + iv(z, y), 
then the Cauchy-Riemann equations yield that 


G! = uz + tv, = vy — iy 
and 
u = . 
G" = Uge + Wag = —Uyy — Wyy 


and we see that u and v satisfy the Laplace equation by equating the real and imaginary parts of G’. Our work in 
previous sections shows that these harmonic functions are C'~-smooth, and it follows that the real and imaginary 
parts of fare C'-smooth and satisfy the Laplace equation and the Cauchy-Riemann equations. We may now use 
Green's theorem on a disc to prove the Cauchy theorem for f on discs. Hence, fis analytic-circ, and we have 
proved the equivalence of the definitions, revealing that fis locally the uniform limit of complex polynomials and 
is given by the Cauchy integral formula. The shortcuts we have revealed in the theory of analytic functions deliver 
us to page 114 of Ahlfors. 

Before we conclude this section, we present one last alternate way to define analytic functions that might be of 
interest to experienced analysts. The result is known (see, for example, Springer [8] or Globevnik [5]), but we are 
in a position to prove it rather efficiently here. We now will define a continuous complex valued function to be 
analytic-ave on the unit disc if f(z)(z— a) satisfies the averaging property on circles C,,(a) contained in the disc, 
Le., if 

20 


0= f(a + re”) (re) do 
0 


whenever the closure of D,(a) is contained in the unit disc. Note that this condition is equivalent to the condition 
that 


0= / fdz 
C,(a) 


for each such circle. We will now prove that analytic-ave functions are analytic in the usual sense. This result can 
be viewed as a version of Morera's theorem saying that a continuous complex valued function that satisfies the 
Cauchy theorem on circles must be analytic. 

Let x(t) be an real valued non-negative function in C'®(0, 1] that is equal to one for t < + and equal to zero 
for t > 3. Define 


(2) = ex(l2!*) 
where cis chosen so that i odA = 1. Let @, denote the approximation to the identity given by 


be(z) = w(z/€). 


The proof of our claim rests on a straightforward calculation that shows that 
ZX be(z —w) 
is (z — w) times a function ~,(z — w) that is radially symmetric about z in w. In fact, 
we(z) = £x'(|2|?/e). 


The calculation hinges on the chain rule plus the fact that 


a — 2 (zz) =; 


(d) 
Oz 


Given a continuous function fon the unit disc such that f(z)(z — a) satisfies the averaging property on circles 
C,(a) compactly contained in the disc, let fe = dc*f for small € > 0. Note that f-is C® smooth on D;_,(0) and 


that f, converges uniformly on compact subsets of the unit disc to fas € + 0. One can differentiate under the 
integral in the convolution formula to see that 


Ofe x. Pe % =f Ute z—w)f(w)dA 
Oz Oz ii weD,(0) a )F( ) ? 


and the observation about the radially symmetric function and our hypothesis about fallows us to write the integral 
in polar coordinates about z to conclude that f, satisfies the Cauchy-Riemann equations, and so is analytic-diff on 
D,_,(0), and consequently, is analytic-circ there, too. It is easy to see that uniform limits of analytic-circ functions 
are analytic-circ. We conclude that fis analytic-circ and so analytic in any sense of the word. 


——————————— 
1.7. The Dirichlet Problem in More General Domains 


We solved the Dirichlet problem on the unit disc, given polynomial boundary data, by explicitly extending 
individual terms z”Z™ as harmonic polynomials. Another way to approach this problem is via linear algebra. 
Suppose a domain © is described via a real polynomial defining function r(a,y) (meaning that 
Q = {x + ty : r(a,y) < 0}), where r(x, y) is of degree two. For to be a bounded domain, it is clear that the 
boundary of 0, which is the zero set of r, must be a circle or an ellipse. Let A denote the Laplace operator. Now, 
the map ¥ that takes a polynomial p(z, y) to the polynomial A(rp) maps the finite-dimensional vector space Py 
of polynomials of degree N or less to itself. (Multiplying by r increases the degree by two, and applying the 
second-order operator A brings it back down by two.) We claim that the map ¥ is one-to-one on Ay, and 
therefore onto. Indeed, if A(rp) is the zero polynomial, then rp is a harmonic polynomial that vanishes on the 
boundary. The maximum principle implies that it must be the zero polynomial. Consequently, p must be the zero 
polynomial, and this proves that ¥ is a one-to-one linear mapping of a finite-dimensional vector space into itself, 
and so also onto. Now, to solve the Dirichlet problem on Q, given polynomial boundary data q(x, y), we know 
there is polynomial p such that A(rp) = Ag. The polynomial g — rp is harmonic on Q, and equal to q on the 
boundary. It solves the Dirichlet problem. We could have solved the Dirichlet problem on the unit disc in the realm 
of polynomials without ever writing a formula down! Now we can solve the Dirichlet problem on an ellipse using 
the same procedure that we did on the disc. (However, the next obvious step, to try to write down a Poisson 
integral formula on the ellipse, gets more complicated because the monomials are not orthonormal in the boundary 
inner product of the ellipse.) 

The Khavinson-Shapiro conjecture states that discs and ellipses are the only domains in the plane having the 
property that solutions to the Dirichlet problem with polynomial data must be polynomials. It is tantalizing that it 
seems so much harder to settle this question than the same problem with the word “polynomial” replaced by 
“rational.” Only discs have the property that solutions to the Dirichlet problem with rational boundary data must be 
rational (see [4]). 

For the remainder of this section, we will let our Poisson and Dirichlet urges run rampant and explain how the 
ideas in the previous sections might be used to solve the Dirichlet problem on more general domains. We will 
dispense with proofs and follow a line of bold declarations. The interested reader can find a more sober exposition 
of some of these ideas in Chapters 22 and 34 of [2] and in [3]. 

To solve the Dirichlet problem on a domain bounded by a Jordan curve, one can use Carathéodory's theorem 
(the theorem that states that the Riemann map associated to such a domain extends continuously to the boundary 
and maps the boundary one-to-one onto the unit circle) to be able to pull back solutions to the problem on the unit 
disc to the domain. But we wonder if there might be a more elementary way to do it. 

Gustafsson's theorem [6] states that a bounded finitely connected domain with n continuous simple closed 
boundary curves can be mapped to an n-connected quadrature domainQ with smooth real analytic boundary via a 
conformal mapping that is continuous up to the boundary and as close to the identity map in the uniform topology 


of the closure of the domain as desired. Such a “nearby” quadrature domain has the property that the average of an 
analytic function over the domain with respect to area measure is finite linear combination of values of the function 
and its derivatives at finitely many points in the domain. The resulting “quadrature identity” is the same for all 
analytic functions that are square-integrable with respect to area measure on the domain. Smooth real analytic 
curves have “Schwarz functions” $(z) that are analytic on a neighborhood of the curve and satisfy S(z) = Z on the 
curve. The Schwarz functions associated to the boundary curves of our quadrature domain Q, have the following 
stronger properties. There is a function S(z) that is meromorphic on an open set containing the closure of ( that 
has no poles on the boundary of Q and satisfies the identity Z = S(z) on the boundary. Quadrature domains can be 
thought of as a generalization of the unit disc (which is a one point quadrature domain), and Gustafsson's 
conformal mapping as a generalization of the Riemann map in the n-connected setting. 

Given a continuous function @ on the boundary of our quadrature domain ©, we can approximate it by a 
rational function of x and y via the Stone-Weierstrass theorem since the family of such rational functions without 
singularities on the boundary forms an algebra of continuous functions that separates points. Writing such a 
rational function as a rational function of z and Z and replacing Z by S(z) produces a meromorphic function on a 
neighborhood of the closure of 0 that has no poles on the boundary and that agrees with the given rational function 
on the boundary. If we can solve the Dirichlet problem on 0 with boundary data 


(z—a)" 


for fixed ain (2 and positive integers n, then, by subtracting such solutions from the data, we would have harmonic 
functions that vanish on the boundary and have general pole behavior at z = a. We could then use these functions 
to subtract off the poles of our meromorphic function and obtain a solution to the Dirichlet problem with the given 
rational boundary data. Then we could take uniform limits and solve the Dirichlet problem for continuous 
boundary data @ just like we did in the unit disc. 

In case © is simply connected, it is possible to solve the Dirichlet problem with boundary data (z — a)” using 
a Riemann mapping function. Let f : 2 + D1(0) be a Riemann map. The Green's function G(z,w) for Q is a 
constant times 


and derivatives 


fo G(z,w) 
are harmonic on 92 — {w}, are continuous up to the boundary in z and vanish on the boundary in z, and the 
singularity at wis precisely of the form a constant times (z — w) ™. One does not need to know that the Riemann 
map is continuous up to the boundary to see that these functions extend continuously up to the boundary in z and 
vanish there. This follows from the fact that conformal mappings are proper mappings: the inverse image of a 
compact subset of the unit disc is a compact subset of 0. 

Hence, in the simply connected case, we have a method to solve the Dirichlet problem rather analogous to the 
method we used in the case of the unit disc. There is something appealing about taking a close approximation to 
our original domain, followed by a close approximation to the boundary data, to be able to find an elementary 
formula for the solution to the Dirichlet problem. 

Riemann maps associated to simply connected quadrature domains can be expressed as rational combinations 
of z and the Schwarz function, so solutions to the Dirichlet problem with rational boundary data can also be 
expressed as rational combinations of z and the Schwarz function! 

Another way to construct the Poisson kernel is to express it in terms of a normal derivative of the Green's 
function, which, on a simply connected quadrature domain, is also expressible in terms of a Riemann map, and 
hence, also expressible in terms of z and the Schwarz function. It follows that the Poisson kernel of a simply 
connected quadrature domain is expressible in terms of z and the Scwharz function. Could we do similar things in 
the multiply-connected setting? Could we use Ahlfors maps in place of a Riemann map? Might the Poisson kernel 
there be expressible in terms of z and a Schwarz function and the harmonic measure functions? We wonder. 
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2.1 Introduction 


One of the fundamental constructions for a region in the complex plane is the Cauchy integral—it 
has many applications. In one complex variable, for instance, it can be used to show that a function 
satisfying the Cauchy-Riemann equations can be represented locally as the sum of a power series. In 
several variables, it can be used to show in iterated fashion that holomorphic functions on non- 
pseudoconvex regions always extend holomorphically to larger regions. This is the Hartogs extension 
phenomenon of 1906. There are alternatives, though, to generalizing the Cauchy integral to several 
variables that do not involve iteration. Often they and their related operators are characterized as 
canonical (e.g., Szegd), universal (e.g., Bochner-Martinelli), or geometric (e.g., Cauchy-Fantappie). 


Such constructions are foundational to the method of integral representations which is important 
for proving regularity of solutions to partial differential equations in several complex variables. That 
method (including the applications and all kernel functions discussed here) is described well in the 
Krantz contribution to this volume [24] as well as [23, 26]. 

The present article should be seen as an additional perspective on the method where the kernels 
themselves are considered geometrically and computationally. In particular, we present one of the 
alternatives as a natural cousin to the Cauchy integral. The Cauchy-Leray integral, in particular, is 
constructed at once as a Cauchy-Fantappié integral associated to domains that are weakly linearly 
convex (as defined in §3) and have twice differentiable boundary. It is hoped that this special 
relationship between the Cauchy and Cauchy-Leray integrals might be an avenue by which insights 
for one variable leads to new insights for several variables. 

Our study begins in one complex variable and explores similarities between the Cauchy integral 
and Szeg6 projector. We include examples not only to show the similarities, but also to illustrate how 
hidden symmetries can lead to calculations that suggest a general behavior. Specifically we exploit 
the fact that the Cauchy integral has Mobius symmetry where the Szeg6 projector has holomorphic 
symmetry. 

We next introduce the kind of convexity needed to construct the Cauchy-Leray integral. By 
default, all regions in one complex variable have this property and the Cauchy-Leray integral 
simplifies to the Cauchy integral. (Such simplification is common to integrals of Cauchy-Fantappié 
type!) We then introduce the Fefferman measure and show that if the Cauchy-Leray kernel and Szeg6 
kernel are defined with respect to Fefferman measure, then they have the same Mobius symmetry and 
holomorphic symmetry as in the case of one variable. We conclude by introducing a domain studied 
recently by Barrett and Edholm where the L?-norm of the Cauchy-Leray operator can be found 
explicitly [2]. The details of this higher dimensional case are outside the scope of this article, but 
there is a resemblance to some of the examples in one variable and the unit ball. 

We mention that many of the constructions discussed in this article also appear in the 
contribution to this volume by Krantz [24]. The emphasis here, though, is on making further 
connections between complex analysis in one and several variables and on enhancing the collection 
of examples. 

The Cauchy-Leray integral has been studied in other special cases, too. Barrett and Lanzani 
studied the essential norm for the case of convex Reinhardt domains generally and for the case of L?- 
balls specifically [4]. As well, Barrett clarifies the vector bundle context in which the Cauchy-Leray 
kernel arises naturally. This leads to an interpretation of the Cauchy-Leray integral as a way to 
quantify a pairing between dual hypersurfaces in complex projective space [3]. 


—SSSSS 
2.2 Szego kernel and Cauchy Kernel Under Transformation 

In general, we assume (2c C is a bounded region with C’'® smooth boundary. Most of the 
discussion applies to more general regions, too, as will be apparent in examples. 

2.2.1 Szeg6 kernel 


To begin, the Lebesgue space L?(022) is defined with respect to arc length measure using the 
Hermitian inner product 


Man = f ghds. 
On 


The Hardy space H?(02) is loosely the subspace of functions that extend holomorphically to the 
region ©. Such extensions are unique by the Cauchy integral formula. To be precise, H?(0.) is the 
closure in L?(02) of the space of functions A°®(J) that arise as boundary values of functions 
holomorphic in Q and smooth in Q. Since H 2(02) is closed, one has an orthogonal projection 
S : L?(OQ) + H*(OM) called the Szeg6 projector. 

By the Riesz Representation Theorem, one has that the Szeg6 projector is an integral operator. In 
particular, an estimate based on the Cauchy integral formula shows for each z € 2 that the point 
evaluation g € H*(02) — g(z) is a continuous linear functional. So there is a unique element 
S, = S(-,z) € H?(O2) such that 7g(z) = (9, Sz)gq for g € L?(0M). Alternatively, the Szegé 
kernel can be constructed by taking an orthonormal basis {¢;} for H?(0) and demonstrating 
convergence of the series S(w, z) = S> ¢;(w);(z). With both approaches one has readily that 

j 


S(w, z) = S(z,w). 

The second approach is helpful for computing the Szeg6 kernel in the case of a highly symmetric 
region like the unit disc A = {z: |z| < 1}. By Taylor's theorem, a holomorphic function on A is the 
sum of its Maclaurin series. So a basis for H?(02) is {Wj}... where w;(z) = 2). In fact, this is an 
orthogonal basis and upon normalization one has 


j20 


(2.1) 


A third approach to the Szeg6 kernel is via the Greens function. For the case of a simply 
connected region, one can define 


0°G(w,z 
S(w,2) = /-a aS 


On? 


and subsequently verify the reproducing and orthogonality properties. There is a messy 
generalization of this for multiply connected regions that involves the associated harmonic measures 
of the region. Such formulas highlight that the Szegé projector and kernel belong to the complex 
analytic “canon” of a given region although the derivations are indirect and typically given via the 
corresponding theory of the Bergman kernel, see [5, 14]. 

For our purposes, we consider the effect of a biholomorphism f : 2; — Q2 on the Szeg6 
projector and kernel. Given that 2, and (2 have smooth boundaries, it is known that f extends 
smoothly to 2, f does not vanish on 2, and f is the square of a holomorphic function (see [5], p. 
42). This leads to an isometry Ay: L?(02) > L?(821) given by g > (g’f)\/f’ that also 
preserves Hardy spaces Ay : H?(02) + H*(02). It, too, yields the transformation law 


Si(w, 2) = f"(w)"? So(f(w), f(2) F(2)?. 
(2.2) 


Notice that the reason for the isometry is due to how arc length is affected by a holomorphic map. If 
ds;, ds are the differentials of arc length on 092), O12, and if z(t) parameterizes 092,, then 
(f° z)(t) parameterizes ON with ds; = |z'(t)| dt and 
dsy = |(f°2)'(t)| dt = |f((t))| |2O| at = |f"| dsr. 

The transformation law enables one to write the Szeg6 kernel for a simply connected region in 
terms of any biholomorphism to the unit disc. In particular, if Q is simply connected and f : Q— A 
is biholomorphic, then the Szeg6 kernel for 0 is 


(2.3) 


2.2.2 Cauchy kernel 


In contrast to the Szeg6 projector, there is the explicit Cauchy operator defined initially for 
g € L'(O2) according to 


€9(2)= a; f sen 


Q Wz 


for z € 2. From this definition, one sees that @g is holomorphic in ©. By restricting to functions 
g € C™(OM2), one can show that the holomorphic function @g extends smoothly to boundary. That 
is, @ restricts to an operator C® (02) + C'™(AM). In fact, Y is bounded on L?(02) and therefore 
extends continuously to a bounded projector @ : L?(02) > H?(0). (The reason that @ 
reproduces functions in the Hardy space is the Cauchy integral formula.) In this way, the Cauchy 
projector arises as an explicitly defined alternative to the Szeg6 projector. 

For further comparison with the Szeg6 projector, observe that the Cauchy projector can be 
expressed via @9(z) = (g, Cz) 9, where 


(2.4) 


and Tw) is the positively oriented unit tangent vector at w € O02. 

It is apparent from these representations that the Cauchy projector and kernel will behave well 
with respect to affine transformations f(z) = az+ b for a,b € C. In fact, they behave well with 
respect to the larger class of Mobius transformations f(z) = (az + b)/(cz+d) for a,b,c,d € C, 
ad—bc=1. In particular, for a Mobius transformation f: 2, — 2, one has the same 
transformation law for the Cauchy kernel as for the Szeg6 kernel, 


Ci(w, z) = f'(w)? C2(F(w), f(2)) £12)". 
(2.5) 


This is a direct calculation that uses the fact that the tangent vectors at w € 0921, f(w) € OM, are 
related according to the chain rule by 


T(f(w)) = Fem T(w) = 244 Tw), 


cw 


Naturally, the kernel transformation laws imply operator transformation laws; specifically, (2.2) 
implies ~%,° A; = Ay’ for a biholomorphism, and (2.5) implies 6° Az = Af* G2 for a Mobius 
transformation. 


2.2.3. Examples 


From the Szeg6 kernel transformation law and the fact that the Szeg6 kernel for a disc is known, it is 
immediate that the Szeg6 projector can be expressed in terms of the biholomorphism to the unit disc. 
Given its Mobius invariance, the Cauchy operator, too, should be computable for regions with 
enough Mobius symmetry. We consider three such examples bounded respectively by two circles, 
two lines, and two logarithmic spirals as in Figure 2.1 The behavior of Mobius transformations with 
respect to circles and lines is familiar, but they are well-behaved with respect to logarithmic spirals, 
too. In particular, logarithmic spirals have constant curvature and a natural parameter when studied 
invariantly with respect to Mobius transformation. 
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FIGURE 2.1 


(i) Annulus, (11) strip, and (iii) logarithmic sector. 


Example 1. The Szeg6 kernel for an annulus 2 = {z € C: r < |z| < 1} can be computed the 
same way as for a disc. In particular, holomorphic functions on an annulus can be written as the sum 
of a Laurent series, so a basis for H?(02) is {Vi} jez, where pj = z), This basis is orthogonal and 
upon normalization one finds 


To show the utility of this representation, we compute the Szeg6 projector applied to a function 
g € L?(02) supported on the outer circle. Let 


if w = e'? 
0 if w = re’?, 


Then for z € 2 


a of, en ikO ok : 
= ij : = z 
ae = / “ 1+ r2k+1 dO = aT 


Meanwhile, the Cauchy projector, too, is easily computed. In particular, 


Ti(oi0 »\) — 1 _ie® _ 1 > k —ikO 
Cle 2) 2mi eiP_z Qn Be 


so for z € 2 one has @g(z) = -4f Eee ke do = zi, 
k>0 


Example 2. For the strip 2 = {z € C: 0 < Imz < z} one has a biholomorphism f: Q— A 
given by 


A computation using (2.3) shows therefore that 


S(w,z) = qk esch35¥ 


As for the annulus, we show how this can be used to compute the Szeg6 projector applied to a 
function g € L?(092) supported on the real axis. Take a > 0 and let 


oye (x2 +02) ifw=2+i0 forzeR 
0 ifw=a2-+in for cER. 


Then for z € 2 


ee 1 1 Uz 
SG(z) =, (45 esch 5+) dz. 


«© #2+a? 


Standard methods of using residues to evaluate contour integrals are helpful for this integral. The 
poles for the integrand are at = tia and x = z+ 2mik for k € Z. Then by integrating over 
squares with increasing side length Imz+(2k+1)a for k =0,1,2,..., and centered on the line 
x = Rez one finds after applying the residue theorem that 


is (—1)" 
SG(z) = qoesch**= + 
: : 2, (z+ 2mik)? + a2 
The Cauchy projector is done likewise but one can integrate over the boundary of upper half discs of 
increasing radii. The result for z € 2 is 


+00 
1 i. 2 1 1 
€g(z) = i Siacqe an eee) = Sau ee 
We mention that calculating the Szeg6 projector in this case is unremarkable. Indeed, in the presence 
of a biholomorphism f : 2 — A, the Szegé projector for the strip can be represented as a conjugate 
to the Szeg6 projector for the disc. That is, Aq = As’ La° A;* where A, : L?(0A) — L?(02) is 
the isometry from the earlier subsection. 


Example 3. For the logarithmic sector Q={z¢C:Im(z'*")>0} one has a 
biholomorphism f : 2 — A expressed using the principal value of a complex power via 


1+mi_; 
f(z) = = . 


In this case, 


Tae mi/2 5—mi/2 
S(w, z) = vin a Aa tae 
As for the strip, the Szeg6 projector is understood immediately through a biholomorphism 
f:Q— A. Again, Zo = As’ Sa" A;* where A; : L?(OA) > L? (OQ) is the isometry from the 
earlier subsection. The Cauchy projector, meanwhile, is best computed after change of parameter. 
Using a constant speed parameter, the boundary spirals have parameterizations w, given by 


Q 
Lary 


€ 


t>0-> wy (t) = +¢1-7% 


In fact, the parameter fis related to the arc length parameter via t = s// 1+ 7?. Consider a function 
g € L?(02) supported on the boundary spiral w,. Then for z € 2 


° w', (t) d 
612) =a | GMO Ds 


After switching to an inversive arc length parameter this integral is computable like for the strip. 
Take u = logt so that 


+00 
=i ° uy etm) du 
€g(z) = — / (g w+ )(e ) oul ae 
(Typically the inversive arc length parameter for these spirals is given as u = ./7 logs; here any 
multiple will suffice.) To show the utility of this representation consider g defined for a > 0 by 
one (u? + a)! if w = w,(e") for ue R 
0 if w = w_(e") for uE R. 


Again using a contour integral approach, one finds poles for the integrand at u = tia and 


u = (log z + 27ik)/(1 — mi) for k € Z. Restricting to the poles in the upper half plane and applying 
the residue theorem gives 


-++oo 1 . 
1—mi et) du 
oe) - 4 fs 
—oo 


u2 + a2 eu(l—mi) x 


fy es eae 5. 1 
~ Qia — eia(1—ni) log 2-+2mik \2 
: fag ee 


where the prime on the summation means to include only k for which Im ((log z+27ik) /(1-7i)) > 0 


. For z = e'* t!-™ with a € (0, 7) this means k > |= aa et | = 


2.2.4 Notes 


Our definition of the Hardy space is the one used in Bell's book [5]. The classical definition says 
that a function is in the Hardy space provided it is holomorphic and the supremum of its averages 
along curves approximating the boundary from the inside is finite. For the equivalence of these 
definitions, see [5, p. 17]. 

As outlined in the Krantz contribution [24], the Szeg6 and Cauchy projectors are related via 


def ‘ 
S=EC(F+LH a where the imaginary part of the Cauchy projector a = @ —@ is the 
Kerzman-Stein operator. This leads to the integral equation 


ee / =(t@) - TO) 5 (a) de, = 040 


giving the Szeg6 kernel as the unique solution to a Fredholm integral equation of the second 
kind. See [5, 22]. 

From the last note it is apparent that @ = Y/ only if & =0. Kerzman and Stein give an 
elementary geometric argument in terms of the kernel to show that this happens only for a disc or 
half-plane. See [21]. Using Taylor expansions of the kernel, one can show that this result persists 
when one uses measures besides arc length. See [9]. 


Our examples show how the Szeg6 kernel can be expressed in terms of the Riemann map. 
Conversely, Kerzman and Trummer show how the Riemann map and Ahlfors maps can be 
recovered numerically from the Szeg6 kernel after solving the Kerzman-Stein integral equation. 
See [13, 22, 28]. 
Using the identity ||@|| = \/1 + |]./]|?, one can find 

— ||@|| = V1 +r for Example 1, 

— ||@|| = V2 for Example 2, and 

— ||@|| > coth(1/2) for Example 3. (See [8].) 


Other example regions for which the Cauchy and Kerzman-Stein operators are at least partially 
computable include the wedge and ellipse. See [8, 10, 12, 16]. 

The Szeg6 kernel transformation law (2.2) extends in a simple way for the case of a proper 
holomorphic map provided the target region is simply connected. If the target is multiply 
connected, the transformation law involves derivatives of the associated harmonic measure 
functions. See [15, 19, 20]. 


ee 
2.3 The Cauchy-Leray Operator 


We now introduce a generalization of the Cauchy operator appropriate for convex domains in high 
dimensions. Take n > 2 and let D CC C” be a domain with C smooth boundary and defining 
function r. This means that D = {z¢ C”:r(z) <0} for a continuously twice differentiable 
function r: C” > R with Vr|ap # 0. 


2.3.1 Convexity and weak linear convexity 


Our understanding of a convex domain is one supported by hyperplanes tangent at the boundary. In 
particular, the tangent space at any point of the boundary does not intersect the domain. In complex 


coordinates this is written Re(Or(w),w—z)>0O for w=(wi,...,wn) € OD, 
z= (21,.--,2n) € D. The brackets should be interpreted as 
". Or 
(aw — : = d. Buus (w; — 2;); 


this is the natural pairing at w of the (1,0)-form Or with vector w — z. Our specific construction of 
the Cauchy-Leray operator allows that D has a weaker geometric property called weak linear 
convexity that can be expressed as (Or(w), w — z) # 0 for all w € OD, z € D. This means that the 
maximal complex subspace of each tangent space at the boundary does not intersect the domain. The 
difference between these conditions can be seen in the following examples. 


Example 4. Let D= {z €C*:2Rez + |z2|? < O}. Then we€ OD, z€D mean that 
wy + W1 + WeW2 = 0, 21 + 21 + ZZ < 0. From this we estimate 


2Re (Or(w),w—z) =2Re(1(wi — 21) + Wo(we2 — 22)) 
=w1it+ WwW — 24 — 21 + 2weWe2 — W2z2 — W2Z2 


> WoW. + 2222 — W222 — W222 
= 2 

= |w2 — 22| 

>0 


It follows that Dis convex, and therefore, also weakly linearly convex. 


Example 5. Let D= ie €C?:-142Rezi < O}. Then we€ OD, z€D mean that 
—1+ wi ew, =0,—-1+ a + Ze < 0. From this we estimate 


(Or(w), w — z) = 2wi(wi — 21) + O(we — 22). 


So (Or(w),w — z) =0 means that w; = 0 or w; — z; = 0. The former case is impossible since 
2Re wr = 1; the latter case is impossible since 2 Re wr =1 and 2Re a <1. It follows that 
(Or(w), w — z) £0 and Dis weakly linearly convex. 

To see that D is not convex, it is simplest to recognize that D is a product of a region bounded 
inside a hyperbola (in the z,-plane) with a complex plane (in the z,-plane) as in Figure 2.2. To check 


the claim algebraically take w = (wi, w2) € OD with w; = u+ iv, we = 0 so that u? — v? = +. 


Subsequently take z = (21, 22) with z; = * +10, zo = 0. Provided u? > 5, then z € D since 


-14+2Re2? = -142(2)' = 41 <0. 


FIGURE 2.2 


Product domain that is weakly linearly convex. 


Meanwhile, 


Re (dr(w), w z) = Re (2w1(w1 — 21)) = Re (2(u + iv)(“ +iv)) =0. 


2.3.2 The Cauchy-Leray operator 


For a smooth, bounded, weakly linearly convex domain D, the Cauchy-Leray operator applied to a 
continuous function on the boundary is given by 


n-1 


m= 1 Or(w)A(O0r) 
£9(2) ~~ (2zi)" a g(w) (Or(w),w—z)” 
(2.6) 


for z € D. The hypothesis of weak linear convexity implies that the denominator is nonzero for 
z € D. The additional hypotheses of boundedness and smoothness of D imply that both the (2n — 1) 
-form in the numerator is a continuous multiple of surface measure and the denominator is bounded 
away from zero (for fixed z € D). These ensure that the integral is finite. Since also the integrand is 
holomorphic in z and the boundary is finite it follows that g is holomorphic in D. 

It is a harder fact that the Cauchy-Leray operator reproduces holomorphic functions in D. In 
particular, if gis a continuous function on OD that extends continuously to D and holomorphically in 
D, then #g = g. The usual proof of this for convex domains involves writing the Cauchy-Leray 
kernel as a Cauchy-Fantappié kernel using a generating form based on the pairing (Or(w), w — z). 
The argument is outlined in the Krantz contribution [24]; for more details, see also [23, 26]. 

It is important to note that the Cauchy-Leray operator reduces to the Cauchy operator in case 
n = 1. This can be seen easily by canceling the common non-zero factor Or/Ow in the numerator 
and denominator of (2.6). Notice that the maximal complex subspace of the line tangent at the 


boundary is the point of the boundary itself. So all smooth regions in C are weakly linearly convex. 
An equivalent formulation of the Cauchy-Leray operator is due to Stanton and involves the Levi 


determinant 
i TZ 
J(r) = —1- det : 
r j Y ik 


Here the subscripts refer to partial derivatives; i.e., r; = Or/Ow;, rz = Or/OWs. If do™® is the 
surface measure on OD (induced from €*), then 


(2.7) 


for z € D. 

We point out that although the Cauchy-Leray operator is presented in terms of a defining 
function, the operator does not depend on the choice of defining function. For this, consider the 
second formulation and take defining functions r, 7 related by 7 = hr for a positive function A. From 
the product rule, we find that if w € OD (and so r(w) = 0) then |V7(w)| = h(w)|Vr(w) 


(OF(w), w — z) = h(w)(Or(w), w — z), 


and using elementary linear algebra, 


t = det 


hr hr; 
= det 
hr; Ar x 


Combining these gives 


which verifies the claim. 


Example 6. To _ illustrate the two formulations, consider the unit ball 


def 


D={2ec? in)" zi|? + |z2 


Pas 1 of. It is straightforward to check that 


£G(z) = 


1 | g(w) (W dw, x dw» /\ dwy + Wodwy A dw, \ dw) 
(2m)? oD (1 — W121 - W222)" 
Meanwhile, J(r)(w) = 1 and |Vr(w)| = 2 when w € OD. So then 


g(w) da’ 
Y5q(2) = gy | Se __ 
oD (1 — W121 - W222) 


To see that Y = Y°* it is enough to check that 


do®" = —+(widw A dW. \ dw2 + Wodwye A dw 1 A dw). 


Since dr annihilates vectors tangent to OD, this follows from wedging both sides by 
dr = wi dw, + w)dw, + Wodw. + wodwe and applying the general formula 


dr \ do® = |Vr| dx, A dy; \ dx2 /\ dy2 for an embedded boundary. 


2.3.3 Notes 


¢ Our definition of weak linear convexity follows that of Andersson, Passare, and Sigurdsson [1, p. 
16]. For the fact that any weakly linearly convex domain is pseudoconvex, see Proposition 2.1.8. 
This will be used in the next section to show that J(r) > 0 on OD. 

¢ The Cauchy-Leray operator is known alternatively in the literature as the Leray operator (or 
transform), the Leray-Aizenberg operator, or simply the Cauchy operator for convex domains. 

« Using the Cauchy-Fantappié method, Lanzani and Stein establish the reproducing property for 
the Cauchy-Leray operator in the case that D is strictly C-linearly convex. This condition is 
stronger than weak linear convexity but weaker than convexity. The condition says 


(Or(w),w — z) £0 forw € OD, z € D\ {wt}. See [25, p. 276]. 


———————S—SSs 
2.4 The Cauchy-Leray Transformation Formula 


We begin this section by introducing the Fefferman surface measure that was constructed for the 
purpose of having a transformation law for the Szeg6 kernel under biholomorphic mappings in high 
dimensions. We then establish the same transformation law for the Cauchy-Leray kernel under 
Mobius transformations. We conclude with a summary of the properties of the Cauchy-Leray 
operator that distinguish it from other reproducing kernels in high dimensions. 


2.4.1 Fefferman surface measure 


The Fefferman surface measure do‘ for a smooth pseudoconvex domain D = {z € C” : r(z) < 0} 
is defined as the measure on OD for which 


dak \ dp = i" 2-H (nr) VOY dz, A dzy A+++ \ dn \ din 
_— —2 J (ry dy 


where as in the last section J(r) is the Levi determinant 


r Ty 
J(r) = —1-det 
T5 Vik 


and dv = dx, A dy; \--- A dyn is the volume measure in C”. 
We explain briefly the context in which J(r) is nonnegative on OD; this is needed for J(r)'/ ane) 
to be well-defined. At any point w € OD, one can make a unitary change of coordinates so that 


rj(w) =0,1<j<n,andr,(w) ¥ 0. In these coordinates, 


I(r)(w) = |rn|? det (rg) 


j,k=1 


The latter determinant is the product of the eigenvalues of the Levi form restricted to the complex 
tangent space. For a pseudoconvex domain such eigenvalues are nonnegative and we conclude that 


J(r) > 0 on OD. From the notes of last section it follows that J(r) > 0, too, in the special case of 


weak linear convexity. 


Any constant multiple of do‘ also will be an invariant surface measure. Here, the constant was 


chosen so that the measure coincides with the Euclidean surface measure for the boundary of a ball 
as well as the arc length measure in one dimension. 
With respect to Fefferman measure, we define function spaces L? (OD) using the inner product 


Mon = f ghdo', 
OD 


(2.8) 


If we further assume D, is simply connected, then as in case n=1 a biholomorphism 
F=(f},...,f"):D,— Dy gives rise to an isometry Ap: L?(0D,) > L?(0D,) via 
g->(9 F) K pt ("*)) Where we denote the complex Jacobian by Jp = det( fe ). As before, subscripts 
refer to partial derivatives. Here, the condition on D, ensures that the fractional power makes sense. 


Notice, too, that the isometry preserves holomorphicity. 
The reason for the fractional power in the isometry is due to the transformation of the Fefferman 
measure. On OD, and ODaz, respectively, 


dof \d(r°F) = —2(r° PF) dy 
dofef \ dr = —2J(r) Mn) dy, 


It is standard for a biholomorphism that F’ (dv) = J F i dv. The related identity 
J(r°F) = J(r) - |Jr|? follows from taking the determinant of both sides of 


r FE (r°F), 1 0 « i) 1 0 
Pp (Eee p NOt Nee eee \ OTe 
Since F’ (dr) = d(r° F), we conclude F” (do) = | Fp | dott, 

It follows that if % :L?(0D)— H?(OD) is the orthogonal projection (with respect to 
Fefferman measure) to the subspace of functions that extend holomorphically to D then one has a 
transformation law “4° Ap = Ar’ .%. As before, this Szeg6 projector is an integral operator whose 
kernel satisfies the transformation law 

$1(w, 2) = Jp (w) Sol F(w), F(2)) Jp (2). 
(2.9) 


In dimension one this reduces to the situation of the previous section (where the requirement that D, 
be simply connected was not necessary). 


2.4.2 Cauchy-Leray transformation formula 


We proceed to show that with respect to the Fefferman measure the Cauchy-Leray kernel is given by 


n—L! Jr n/(n+1) w 
C(w, z) = 7 Oru)" 


(2.10) 


and this kernel satisfies the same transformation law as for the Szegé kernel provided F': D; + Do 
is a Mobius transformation. In this situation there is no further assumption about D, since the 


fractional power of J; makes sense. (See item (a) below.) 
For the first claim, we start with the calculation 


ag i. 


1 (Or A (Or 


Omi" )Adr = —4,0r A Or A (O0r 


(27i)” 
= — 21! Jr) du 


wn 


=e Ue T(r) dat \ dr 


on” 


where the third step uses the definition of Fefferman measure and the second step uses 


= x n—-1 
Or \ Or A (O0r) — = —(2%)"(n — 1)! J(r) dv. (The proof of this identity is essentially the same as 
what we used to explain why J(r) > 0.) Since dr acts transversely to the boundary, it follows that 
when restricting to OD and dividing by (Or(w), w — z)”, 


(n—1)! J(r)"/"Y (w) fete 1__ Or(w)A(d0r) 
2m” (Or(w),w—z)" doy = (27i)" (Or(w),w—z)" * 


def 
Referring back to (2.6) and (2.8), we have g(z) = (g,Cz)ap provided C,(w) = C(w, z) with 
C(w, z) as expressed in (2.10). 
For the second claim, we show that if F = (f1,...f") : D1 + Dg is a Mobius transformation 


with f2 =9;/9nur I< Jj <n), g;(w) =ajiwit-::+ajnWntajny I <j<nt+)), and 
det(a;~) = 1, then 


Ci(w,z) = Tp!) (w) C2(F(w), F(z) Tp! (2). 
(2.11) 


Beyond what already is established, the proof requires the following: 
(@) Tp(z) = (1/gn+1(z))"™ 
0) SO (°F) ;(w) (ws — 23) = Dri F(w))(F7(w) — £7(2))9n+1(2)/9n+1(w). 


For (a), 


a ofi\ 1 Las 95 
Ir(z) det ( Ozk Inv det (ax An+1k Jn+1 ) 


Ajjk — An+1,k 93/9n+1 ) 
Qn+1,k 1 


Soi ag jk 9;/9n+1 ae ee ask 95 
In Crh @L TH Gn+ik Gntl 


Ajk = Ajnt1 )= 1 


Antik On4+1,n4+1 


For (b), 


: - _i€FW)) Gnsa 0) ast a1 95(0)/Snea He ~ 24) 
= © nee) grva(tw) *[9(w) — 99(2) — [gna(w) ~ gnea(2)) 202] 
= 5 rj(F(w)) gnvi(w) [gn+1(z) 9;(w)/gn+1(w) — 9;(2)] 
= Dy r(E(w))(f4(w) — fi(2))9n41(2)/Gn1(w). 


With these in hand, if F : D,; — D2 is a Mobius transformation and ris a defining function for D,, 
then 


oot es 2 myn/(n+1) 
Ci(w, z)-2n"/(n—-1)! = ar) “ is 
1( ) /( ) ie (r° F) ;(w)(w;-z))) 


I(r) | D(F(w))-|Jp(w) |?! 1) 


(2 (r°F);(w)(wj-2)))" 


"= J(ry"/" (EF) gnyi(w)” 2n/(n+1) 

— nm ° nm * J ‘W 

(Snireonrw-reyy aac eC 
n/(n-+1) I(r)n/ +1) (F(w)) n/(n+1) 

== J W): nr -J z 
ee (SnFwrw)-rey)y ~* ) 


= Jn) wy) Co(F(w), F(z) Tn!) (z) - 2n®/(n — 1). 


Taking conjugates establishes (2.11). 


2.4.3 Comparison of kernels 


Here we give a brief comparison of some of the more prominent boundary integral operators and 
kernels in complex analysis. These include the ones discussed in this article as well as the Bochner- 
Martinelli and Henkin-Ramirez integrals. For the latter operators, we refer to the Krantz contribution 
to this volume [24] as well as [23, 26]. Of the kernels under consideration, the Szeg6 kernel stands 
alone as a non-explicit kernel—its existence is known but it is known precisely only in special 
situations. 


Beyond this, we have the Table 2.1 to summarize the key differences. Class of domain indicates 
geometric requirements for construction of the kernel. (The degree of smoothness typically is what is 
needed for the construction—twice differentiable for Cauchy-Leray and Henkin-Ramirez but only 
once differentiable for Cauchy and Bochner-Martinelli.) Holomorphic refers to the free variable; in 
particular, an operator with holomorphic kernel projects to a class of holomorphic functions even if 
integration is done over just a subset of the boundary. The r-dependence indicates whether the 
resulting operator depends on the defining function. Finally, invariance refers to symmetries or the 
existence of a transformation law for the kernel. As shown in this article, there is invariance for the 
Szeg6 and Cauchy-Leray kernels provided integration is done with respect to Fefferman measure. 


Cauchy n = 1 Szeg6n >1 Bochner-Martinellin > 1 | Cauchy-Leray n > 1 | Henkin-Ramirez n > 1 
Class of domain General General General Convex Strictly pseudoconvex 
Holomorphic Yes Yes No Yes Yes 
r-dependent No No No No Yes 
Invariance Mobius Biholomorphism Affine Mobius Affine 
TABLE 2.1 


Comparison of reproducing kernels 


2.4.4 Equivalence and example 


We conclude with an explanation why Y = #°% and present an additional example (besides the 
sphere) that has enough symmetry to enable calculation of an L?-norm. 

The equivalence of (2.6) and (2.7) is a consequence of working out the wedge products and using 
standard formulas for surface area in terms of differential forms. To be precise, using the identity 


Or \ Or A (aar)” = —(2%)"(n — 1)! J(r) dv from the earlier section, we have 
=~, \no d 2i)"(n-1)!J 
(Or A (Bdr)” ) A dr & = —(21)"(n — 1)! I(r) 7A p dor & = 2) a ) aa" A dp 


n-1 


1 Or(w)A(00r) _ (n—1)! J(r)(w) do*°(w) 
(27mi)"  (Or(w),w—z)” mn” |Wr(w)| 


That Y = Y°% then follows from (2.6) and (2.7). This is essentially the same proof that Stanton 
gave in [27]. 


Example 7. We conclude with the domain for which Barrett and Edholm compute the L?-norm 
of the Cauchy-Leray transform as a singular integral operator on the boundary [2], 


def 
D=D3= {= (z1,z2) € C? : r(z) = |z1|? + BRe(z?) —Imzp < of 


for 0 < 6 < 1. It is straightforward to check that D is strictly C-linearly convex. (The definition of 
this property is given in the Notes of §3.) In particular, 
(Or(w), w — z) = (Wi + Bwi)(wi — 21) — 3 (w2 — 22). So for w € OD, z € D, 


The first step uses w € OD and the second step uses z € D. There is equality in the second and third 
steps only if z<€ OD and w, = 21, respectively. To finish the claim, it is enough to see that 


(Or(w),w — z) #0 for w,z€ 0D with uy = 21, We # 22. But then, 
(Or(w),w—2z) = + (we — 22) is clearly nonzero. 
Using (2.6) we find that for a function g defined on 0D, 


292) = ge | — ea 
(27i) oD [(wy + Bw)(wi 21) 1 (we z2)| 


2i 


for zeéED. Take Wi = Uz + iv; 25 = 5+ iy; with Uj, V5, L5, Yj; © R so that 
vo = (1+ B)ut + (1 — B)v? and y2 > (1 + 8)x7 + (1— B)y? since w € OD, z € D, respectively. 
In these coordinates the integral simplifies to 
w) dug A du, A dv 
Laz) = ds f = ao : ; 
RY [1 + Bwi)(wi — 21) — a (we — 22)] 


The denominator is more complicated than our earlier examples, but still it is the square of a 
quadratic function. As shown in [2] (and indicated in the notes below), there are hidden symmetries 
that work to simplify the calculations. In fact, the first main result in [2] is the calculation 


1 


L?(D,,o%*) — VS Te . 


Alternatively, one might attempt simpler calculations, for instance, by (i) considering a simplified 
function g (as in Examples 1-3), (ii) restricting to 8 = 0 (so Dis the Siegel upper half-space), or (iii) 
evaluating only at z = (0, iy2) for y2 > 0. 


I 


2.4.5 Notes 


¢ With respect to Fefferman measure, Hirachi proved the Szeg6 transformation formula for strictly 
pseudoconvex domains and for certain strictly pesudoconvex CR manifolds [18]. Fefferman 
introduced the measure in [17, p. 259]. 

Like the case n = 1, the Cauchy-Leray and Szeg6 projectors are the same only when the domain 
is a Mobius image of a ball and only when one uses Fefferman measure. Similar results hold for 
the case of the Bochner-Martinelli kernel. See [6, 7, 9]. 

A more detailed analysis of the Cauchy-Leray operator for the domain Dz in Example 7 leads to 


a representation that is reminiscent of that for the unit ball in Example 6. In particular, [2, $4] 
shows how our first representation of for the ball is the same as that for Dy if one replaces the 


conjugate holomorphic integration variables (w ,w2) for the ball by “projective dual 
coordinates” for Dz. (The specifics of the duality construction are outside the scope of this 


article.) The domain Dy has been studied, too, using methods from differential geometry. See 
[11]. 
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3.1 Introduction 


It was a pleasure to get an invitation from Professor Krantz to add a chapter on some topics from 
function theory to this text that he is putting together. I have assumed that the reader is familiar 
with basic material about complex numbers, analytic functions and maybe a bit of linear algebra 


in the setting of KR” and C”. My goal is to introduce the reader to the basic material on fractional 
linear transformations in the setting of the complex plane. There are many extensions from this 
“scalar” setting to higher dimensional settings. I plan to segue from this complex analytic setting 
to extending some scalar functions (the fractional linear transformations) to operator-valued 
functions and eventually to prove one result about operators that are contractions. I have to leave 
the readers at a point where if they are interested they can get into some extremely useful and 
classical operator theory. I hope the reader will find the presentation readable and if not you 
might try one of the references that I have listed. Any errors in the presentation are my own, but I 
have tried to carefully read over the material with the hopes of culling out any serious “faux pas”. 


——SEE—E—EEE==_ 
3.2 Notations and Definitions 


The letters a,b,c, and d are fixed complex numbers, and the letters Z and W represent complex 
variables. The letter C stands for the complex plane and the notation S = C U(occ) represents the 
complex plane with the “point” at infinity added. This can be realized as the closed unit sphere, 
but we shall not need that information in our work. 


The words “fractional linear maps” stand for the family of mappings f(Z) = aget =W. 
These are also known as Mobius mappings. There are certain special cases (choices of a,b,c, or d 
) that we wish to avoid and let us say a word about this matter. First, if c is zero and dis zero we 
can see there will be serious questions about the meaning of such an expression. To avoid this we 
assume (the determining number) ad — bc ¥ 0. 


For if this were not the case we observe that for different values Z, and Z, the expression 


Z_—Z,)(bc—ad 
f(41) — f(Z2) = REC =O 


implying that f is a constant mapping. So for example the choice of a = d= 1 and b= 
c = 22 would yield 


a 


Oi and 


Z+5 7 
f(Z) = yaa F9°* 


Further, in this case if ad — bc = 0 and assuming c = 0, we have the unpleasant situation that 


a = 0 or d=0. We do not define a f(Z) = “4** = W in these cases. We let the symbol L 


cZ+d 
denote this family of mappings. Note also that if AX 40 then f(Z) = ae 


a,b,c, and d constants are not unique. 


as well, so the 


It is easy to arrive at 


f'(Z) = 2 40 


(cZ+d) 


which implies that fis locally one-to-one (a proof is given later), but of course more is true. We 


can find an inverse for f(Z) = W which is g(W) = ov . Note that a computation shows for f 
as above and g(Z) = Agee the functional composition of g with £ g° f(Z) = W(Z), yields 


another element of L 


__ (aA+eB)Z+(bA+dB) 
= (aC+eD)Z+(bC+dD) 


W(Z) 
and it will also follow that the determining number 


(aA + cB)(bC + dD) — (bA + dB)(aC + cD) = (AD — BC)(ad — bc) £ 0. 


Hence, L is closed under the operation of composition. Further, each f € IL is one-to-one and 
has an inverse under composition in the set L. 

The reader may have noticed that our requirement on ad— bec is reminiscent of the 
determinant of a two-by-two matrix. In fact, it is very useful to identify the element f in 


f(Z) = 445 = W of L with the two by two matrix 


a b 
ae i) 


Note first that if we consider the calculation with the composition of g and fas above, it is 
just the matrix multiplication of the two matrices associated to g and f 


A B\(/a b (aA+cB) (bA+dB) 
@ a ‘ 1) = Co cD) (bC + oy) 

Further, our requirement for non-degeneracy of the determining number is the requirement 
that the determinant of the associated matrix be non-zero (i.e., the matrices are invertible). Since 
matrix multiplication is associative the same holds for L, (h’g)° f =h°(g’ f). Hence, we now 
can view L as a group with invertible elements. Note that since matrix multiplication is not 


commutative the group L is not commutative. Further, if one recalls the easy formula for the 
inverse of the two-by-two matrix 

a b 

c d 


, (d -b 
ad—bc —ca 


and the associated form for the inverse of f(Z) = W is 


then the inverse is just 


—_ 1 dWw—b 
g(W) ~~ ad—be —cW+a 


and it has 1 for it determining number, and this is the same form for the inverse as we have 


calculated above with the scalar \ = (be — ad)". The elements of L play an important role in 
the theory of conformal mappings, one variable operator theory, and several other aspects of 
classical analysis and we shall discuss some of these uses later in this work. 


It is possible to put an equivalence relation on L as follows. 


Definition 1 For each g and fin, we say that g is equivalent to f if there exists an element h € 
L with 


g=h’ f'm, 
where m is the inverse of h. 


It is easy to check that this is an equivalence relation and we shall make use of the fact later. 
Let us view some paradigm examples of elements of the L with their defining numbers. 


1. The identity map f(Z) = Z, with ad — be = 1. 


2. Magnifications, m(Z) = aZ, with ad — bc = a. 
3. Linear mappings, /(Z) = aZ + b, with ad — bc = a. 


4. Inversions, v(Z) = =, with ad — bc = —-1. 


Lt 
Z 


It is very useful to observe that a general element f(Z) = ete = W of Lcan be built up of 
these four maps. Namely, writing 


aZ+b _ a/c(cZ+d)— ad —b) _ a b= 
cZ+d cZ+d ae eged 
we can write W11 = cZ, Wo2 = W,1+d, W3 = wii ... etc. to arrive at the decomposition 


from the elementary maps. 


There is something very useful that comes out of the elementary mappings above about a 
general mapping from L. 

Namely, it is obvious that the identity mapping preserves geometric shapes, in particular lines 
and circles in the plane. Further, we see that translates also preserve geometric figures. The 
magnification map m(Z) = aZ also maps a circle with center P and radius r to a circle with 
center aP and radius ar, and the reader can check that it maps affine lines to affine lines. Hence, 
the first three maps above preserve the families of circles and lines in the plane. More is true in 
that any map from L has this property. So, assuming that we identify lines as unbounded circles 
through infinity and circles in the plane as bounded circles, we have the following result. 


Theorem 1 Every element of L. maps circles to circles. 


Proof: To prove this we have only to prove that the inversion map maps circles to circles. 
First assume we consider the (bounded) circle C = (Z: Z = A+rexp 20), 6 © [0, 27]. 
Applying the inversion formula to these values we find 


1+ A? Z?-AZ-AZ 9 
IZ =T, 


1/Z-A?= 

or in real coordinates with Z = x + iy and —2Re(AZ) = ax + By 

(A?—r?)(x* + y’) —ax— By+1=0. 
And this is the equation of a circle in the plane. 

It may be useful for the reader to check that the circle 
Z*+#Z+tZ+m=0, 

is mapped to the circle 

Z7?4++4Z74+tZ/m+1/m=0, m40, 


and to 


iZ+4#Z+1=0, 
ifm =0. 


Note that if we consider an unbounded circle passing through infinity of the form 
C=[(Z=2+maz] with m a real number, the inversion transformation maps C to the 
unbounded circle of the form 


1 a l-m __ 
atimaz ~— x(l+m?) ut Ww, 


which is the circle in the u,v plane, v = —mu. Similarly, one can show that the image of any 
affine line in the plane is mapped to a circle under the inversion mapping and the discussion is 


left for the reader. More generally, if the equation of the circle is given as tZ+tZ+m=0, 
then the inversion oa takes the circle into the circle into the circle Z ? + +Z + a m #0, 


and if m = 0 it is mapped into the circle tZ + tZ = 0. 


CC ee 
3.3. Some Geometric Considerations 


Assume Z,, Z2, and Z, are three distinct complex numbers. They determine a circle in our sense. 
Then the element T € L of the form 


— (4-41)(Z2~23) _ 
M4) = Gaj@azy = 0 


has some interesting properties. Note that it maps Z, to zero, and Z, to the number 1, and Z; to 


infinity. Thus the circle containing 71, Z2, and Z; is mapped to the circle containing 0,1, and the 


point at infinity. This f(Z) = agit = W is known as the “cross ratio” and is denoted by the 


form (Z, Z1, Z2, Z3). Now it follows from these properties if W,, W2, and W, are three distinct 


complex numbers the equation 
f(Z) = (4, 21, 22, Z3) = (W, Wi, Wo, W3) = g(W), 


maps the circle determined by 21, Z2, Z3, one to one and onto the circle determined by the 
points W;, W2, W3. This follows easily since setting Z = Z; gives the value number zero in the 
left terms and since W, is the unique point in the righthand term giving the value zero it must 


corresponding to Z, etc., for the two remaining terms. 


Note that in the above we have assumed the chosen points were finite complex numbers. If 


one of these points is infinity we can still define the cross ratio as follows. For the case Z, = ov, 
_ (4,-4s) : = : _ (Zh) 
we set (Z,00, Zo, Z3) = Z-Tay? and if Z,= oo it becomes (Z,Z 1,00, Z3) = (ZZ) : 


similarly for the case where Z3, equals oo. 


As an example, consider the f(Z) = “+ =W, 


cZ+d 
f(Z) = (Z,0, 1, -1) = (W, 2, 2, 4) = g(W). This takes the form 


—2(W-2 
f(Z) — Fa — Wong = g(W). 


For this example the one circle determined by 0, 1, —1 is the circle Im(Z) = 0. The second is 


the circle with center at (3, 9/2) and radius r = wee 
The solution is W = aaa 


There are many more geometric properties enjoyed by the cross ratio and we mention one 
more and refer the interested reader to the classic text by Ahlfors [1]. 


Theorem 2 The cross ratio (Z1, Z2, Z3, Z4) is real if and only if the four points |Z;| lie on a 
circle. 


There is an analytic proof given in Ahlfors but for those interested in some geometry consider 
the equation 


Z1-Z2)(Z3-Z Z1-Z Z3-Z 
arg( DaNeZ )=arg( az )-arg( Zaz ) 
and draw some pictures. 
aZ+b 


One can check easily that elements f(Z) = “ za = W do not preserve length between 
points in the plane. However, they have other important properties which we consider. Namely, 
suppose I(t) and I(t) are two differentiable curves in the complex plane, and 
I(t.) = I'2(t,). The tangent lines to curves at the point W, = I'(t,) yield two vectors, say V, 
is a vector based at W, and pointing in the direction of the tangent vector to J), and similarly 
assume V, is a vector based at W, and pointing in the tangent direction at I; (t,). The angles are 
chosen consistently with the orientation of the two curves. The angle between the curves I, and 
I, for the value of t, is the angle, say @, through which V, must be rotated counterclockwise to 
lie in the V, direction. Now for an analytic function f defined in an open neighborhood of the 
point W, we have two more curves given by A;(t) = f(I;(t)). The function f is said to be 
conformal at W, if the angle between A; and A2 at f(W.) is equal to the angle between I, and 
I, at the point in question (and this is true for all such curves). 


Theorem 3 An analytic function f is conformal at every point Z, where f (Z,) is not zero. 


Proof: By assumption f(Z.) =A>0. We show that there is a disk, 


C=[Z: |Z— Z,| <r] on which fis a one-to-one function. Since f is continuous, we may 
choose a disc D = [Z||Z— Z,| < r] with |f(Z) — f(Z.)| < (A/2) for Z © C, and so that 
f (Z)—f' (Zo) < (A/2) for any Z €D. 

Let Z, and Z, be distinct points of D and let y = y(t) be the straight line segment in D 


joining them. Then 


f(Zi) — f(Z2) = [Faz = 


| £ @oyaz - [is ()- # haz > 


“y 5 


AB=7 = 47. — 71 = 4217, =F. 20: 


With this fact we show that every oriented, smooth curve through the point Z, has its tangent 
turned through the same angle, under the mapping f(Z). This implies that the angle between any 
two such curves passing through Z, is preserved. So assume y(t) is such a curve, y(t.) = Zo, 


and (to) is the tangent vector at the point in question. Now consider the image curve given by 
f(y(t)) = T(t). I''(tc) = f'(Zo)y (to). Note that our assumption on f(Z,) and the 
smoothness of y implies that I"'(t,) 4 0. Hence, the angles which tangent vectors y'(t,) and 


I'(t) make with the horizontal direction are related by the equation 


arg(I” (to)) = arg f (Zo) + arg 7 (to): 


Thus every curve through Z, is rotated through the same angle arg f (Z,) which is a constant 
(independent of the curve under consideration). 


3.4 The Symmetry Principle 


In the plane (IR*) given a point P = (x, y) (assume with out loss of generality that y > 0) there 
is a point naturally associated to P, namely the reflection of P in the x-axis. That point is 
PQ = (a, —y). The x-axis is orthogonal (at right angles) to the segment joining P to PQ (and 
bisects it). Note also, that if C is any circle in the plane containing P and PQ, its center must lie 
on the real axis. Consequently, this circle must meets the x-axis at two points x1, and x2. The 
angles at which the tangent lines to the circle at points 7;, and x,, where it meets the x-axis, are 


also ninety degree angles. That is, the circle meets the x-axis orthogonally. A picture will help at 
this point. 

In a more general setting, we wish to define the idea of symmetry for any two (finite) points 
in the complex plane. So assume {Z;, 7 = 1,2} are two distinct points in the plane. Let U be the 
line segment joining them, with midpoint of U being u. Assume the slope of U is m. Then the 
line G through u with slope = is orthogonal to U. We say the points Z; are symmetric in the line 
G. Again, since if we choose any circle C through Z;, and Z,, we see that U is a chord of this 
circle and that the chosen circle C will meet G at exactly two points, say P, and P,. The angles 
formed by the tangent lines to C at P,; and P, will be ninety degree angles. That is C meets G 
orthogonally. 


Definition 2 Two points Z; and Z, are said to be symmetric with respect to a line (circle) E if 
every circle through them intersects E orthogonally. 


Theorem 4 Let E be a circle and let f € L. Then two points Z, and Z2 are symmetric in E if and 
only if the images f(Z,) and f(Z_) are symmetric in the circle f(E) = E’. 


Proof: Consider the distinct points W; = f(Z1), W2 = f(Z2). Let V be a circle containing W, 
and W, with V = f—1(V’). Then V meets E at two points Q:, and Q). The inverse points under 


f, f ~@s) = Q; lie on V. Hence, Vis a circle containing Z, and Z,, and so the angle formed by 
the tangents to V at these points is ninety degrees. But fis conformal so the angles are preserved, 


implying that the tangent angle to the image circle V is ninety degrees. This proves the 
sufficiency and we suggest the reader supply the necessity. 


Dr 
3.5 The Automorphism Group of the Unit Disc 


One of the important subgroups of the group L is the set of one to one automorphisms of the unit 
disc D= {Z : |Z| < 1}. That is the holomorphic, one to one, onto mappings of D onto D. To 
develop this section it is useful to have a formula that relates two symmetric points in a given 
circle. 

So assume F is a given circle (finite for this development) with center A and radius r. Let wu be 
a given finite point. The points A—r,A-+ ar, and A+, are three points of E. Utilizing the 
cross ratio, 


Z—(A-r)}{ar—r Z-(A-r 
f(Z) =(Z,A—1, 2+, A+) = Faas = Zea 


We have a mapping from E onto the circle given by the real axis. By the symmetry principle 
if (u) is the refection of in E then T((j) is symmetric to T(y) in the real axis. In complex 


notation (recall if Z = x + zy, then Z = x — zy) we have T((p) =F ( pi) 
From the immediate formula above this leads to the equation 


* AT 
Wat) _ (ye) 


(u)’—(A+r) u—(Atr) ” 
If we solve this formula for (uw) we find that 
* = r2 
(uw) =A+ oT. 
Note that 
pe BAY ae 


and so the argument of (j.4) is the same as arg (2) — A). It follows that wand (2) lie on the 
same ray originating at the point A and that 


2 


(ih) SAS a 


To end this section, we prove an important and useful result about this special subgroup of L. 


This subgroup will be denoted by A. 


Definition 3 A one-to-one holomorphic mapping from the unit disc D onto itself is called an 
automorphism of the disc. 


The next theorem will show that all such mappings are in L . 


Theorem 5 A holomorphic mapping f € LL is in A if and only there is a number, c € D and a 
number t, |t| = 1 with 


Proof: First if fhas the given form and Z € D, then it is easy to check that 
|Z — el? = |Z? 4 |cl? — 2Re(@Z) < |1 — eZ|? = 1+ |cZ|* — 2Re(EZ), 


so f maps D into D, and is one to one. Note that f has an inverse and from this f will map D onto 
D and the unit circle onto the unit circle. Hence, f € A. 
Now assume f is in L and f is in A. Then, by the ontoness assumption, there is a point a in D 


with f(a) = 0. By the formula for the symmetric point we see that a* = 4. Now the symmetric 
point of 0 is « and so f(a’) = oo by the reflection principle above. Such an automorphism has 
the form (recall the cross ratio formula) 


_ (4=a)(Z2-=) 
(2)= (2-4 (Z2-a) go 


Now, using the fact that |f(exp(2t)| = 1,t € (0,27), we conclude that there is a number 
A, |A| = 1, so that 


_ ) Z=2) 
WARE A ean 


Thus f has the desired form. 


 ——————————————————————_ Fe 
3.6 Some Linear Algebra 


In this section, we try to proceed to a bit of material, part of which is usually covered in an 
undergraduate course but some of which will be a bit advanced. In giving the background we 
hope that it will be partial review and that examples will fill in the missing pieces. We assume 


you are familiar with the term “vector spaces” and if not you can find the definition in many 


undergraduate texts. Think of R”, or C” as paradigm examples. A vector space V is finite 
dimensional if there are a finite number of non-zero vectors from V, say {v ne for which given a 


n 

vector X in V we may express it uniquely as the sum X = Ss; Cv;, where the C; are from the 
1 

scalar field (in our case usually the complex numbers). If this is not the case we say that V is 


infinite dimensional. For example the polynomials of degree less than or equal to n considered on 
the interval /0,1/ with pointwise addition and scalar multiplication are a finite dimensional vector 
space of dimension (n+1). A basis is given by the set eg k= ON Sec ,n}. However, if we 
consider polynomials on the same interval of all degrees it is a vector space but it is not finite 
dimensional. As another example from basic calculus the Riemann integrable functions on the 
interval /0.1/ 1s also a vector space and it is not finite dimensional. A norm on a vector space is a 
non-negative, real valued function f(°) usually written as f(°) = ||°||, and it has essentially the 


same properties as the absolute value function on the real line. So for example considering C” a 
n 

norm is given for Z = (21,...,2n) as Z 7 = Ss; z; 7, or in the case of the polynomials on 
1 


[0,1] as ||p(t)|| = max{|p(t) |t € [0,1]}. So we have ||z + y|| < ||x|| + ||y|| for vectors x and 
y in V, and with Ca scalar ||C’x|| = |C|||z||. Such vector spaces are called normed vector spaces. 
If, whenever X,, is a sequence in a normed vector space V and if ||X,— X,|| + 0 (sequences 
exhibiting such behavior are called Cauchy sequences) for k and n tending to infinity, we know 
that there is a unique vector X" € V with X,—X* - 0asn tends to infinity then we say V 
is acomplete normed vector space. Further, if the operations in V are continuous in the sense that 
for x and y in V we have the mapping from V x V into V by the addition operation x + y = z 
and the scalar multiplication operation from C x V by (C,2x) — Cz then this a topological 
normed vector space. 


The examples above are topological normed vector spaces. We shall refer to them as normed 
spaces. Note that the example given above of all polynomials on the interval /0,1/1s not complete 
in that the absolute value function |t — 1/2] is the limit of a Cauchy sequence from the 
polynomials. Try to get a picture of how this might happen.This is a non-trivial statement and 
requires proof, which we do not include. Now consider a few more examples. The first is called 
“little P” and is the vector space of vectors consisting of an infinite number of complex numbers 


[e,¢) 
Z = (21, 22, 23, +++) Zn,---) which satisfy the requirement > z; 7 < oo. For a specific example 
1 


[e.<) 
take the terms to satisfy z; = 1/j. The norm is just Z oe > Zj 2 and with this norm it is a 
I 


normed space. The set of polynomials of degree less than or equal to n is also such a space with 
the sup norm we defined above for the space of all polynomials. Finally, a normed space is a 


Hilbert space H if the norm on H is induced by an “inner product”. That is a mapping from 


V x V, usually written as < U,W >, into C which is linear in the first entry and anti-linear in 
the second and for which ||u||? =< U,U > . That is, for U,W, Z © H, and Ca scalar, 


<U+W,Z>=<U,Z>4+<W,Z>, 


and <U,Z>=< 2Z,U>, and <CU,Z>=C<U,Z>. Normed spaces and Hilbert 
spaces are very useful in theoretical aspects of analysis, but we only touch on one of the reasons 
why. Mappings between normed spaces is one of the important aspects of analysis. For example, 
the map on P given by Z = (21, 22, 23,...) > (27, sin z2/2, 23,..) is an interesting mapping 
however it may not be easy to work with. It is what is called a nonlinear mapping because the 
terms in it are not linear. The study of nonlinear analysis is an important and challenging field but 
we are not going into that area. We study another set of mappings of equal importance called 
“linear mappings”’. 


Definition 4 A mapping T on a normed space VY into a normed space V is called linear if for 
vectors A and B in V and a scalarC' € C we have 


T(A+CB) = T(A)+CT(B). 


Note that the vectors in the argument of T are in the space V whereas the vectors in the range 
of T are in W. Some easy examples on say FP are given by the identity 
Z= (21, 22, 23, ---) + T,(Z) = (21, 22, 23,---), Z — T2(Z) = (0; 215.22; 23335) the shift 
operator, Z — (22, 21, 23,---) = 73(Z), or finally Z > (21, 22/27, 23/37, ...) = T4(Z). Note 
that the identity mapping 7; is one-to-one and maps all of F onto itself. In contrast T, does not 
have any of the vectors of the form (C,0,0,...),C # 0 in its range. The linear operator T, on 


the other hand is one to one and maps onto all of F, and finally given the vector 
W = (1,1/2,1/3,...) € 1? you can check that no vector will map onto it under T,. These 
different behaviors are the norm in that the map T; is special. It has what we call an inverse. That 
is, there is another linear operator (on P), say S, and we = see _ that 
S(W) = S(wi, we, w3,..) = (we, wi, w3,..) has the property that S°T3(Z) = T3°S(Z) = Z 
for all Z € I?. We write this inverse as S = T~'. Note that, in the first few sections, I avoided 
using this notation and wrote out the word “inverse” or used a different symbol. The reason for 
this will come up later. 


Definition 5 An operator T from a normed space X into a normed space Y is invertible if it is 
one-to-one and onto. 


So if T is such an operator with T(X) = Y then its inverse is S(Y) = X. It is easy to see 
that S°T(X) = S(Y) = X =1(X), and that now S = T™! is also linear. This follows for 
example if when we choose X;,j=1,2 in X and CeC with T(X,;)=Y;, then 
S(Y1 + CY2) = S(T(X1) + CT(X2)) = S(T(X1 + CX2)) = X1 + CX = S(Y1) + CS(Y2). 


Definition 6 An operator T on a normed space X into a normed space Y is continuous at a point 
X” € X if, whenever X, € X with X,—X° 0, then T(X,)—-T(X 0. 


Note that for such operators continuity at one point implies continuity at all points of X. This is 
easily checked by showing that for T as above it will be continuous at the origin. For if W,, — 0 
in X the mapping W=X*—X is continuous by definition of normed space and so 


T(W,,) = T(X* — X,) = T(X*) —T(X,) > 0, where X,, corresponds with the W,, under the 
given map. Further there is another condition that is equivalent to continuity. 


Definition 7 A liner mapping T as above is said to bounded on X into Y if there is a number 
M > 0 for which 


||T(x)|| < M||z|, 
forallx € X. 


The infimum of all such numbers M is called the norm of the operator T. Now the reader can 
show that the concept of continuity is equivalent to the concept of boundedness for such linear 
operators on normed spaces. All the operators we have introduced above are bounded linear 
operators. T;, and 73 are invertible but 7), and Tj, are not invertible. The operator T’ = 


differentation on the polynomials on /0,1/ is linear but not bounded since with P,,(t) = a one 
has ||P,,|| =1/n > 0, whereas T(P,) = t™ 1? =1. 


There is an idea used in such spaces which is referred to as completeness. 


Definition 8. If every Cauchy sequence {x,,} in a normed vector space X has a limit x in the 
space then X is a complete normed vector space, or for short a Banach space. That is, if 
\|an — 2%|| 4 0;n, & > 00 then ||x,, — z|| > 0,n > oo. 


Having discussed (albeit briefly) the idea of a linear operator on normed spaces we wish to 
conclude with an idea that is useful in operators on Hilbert spaces. 


pl 
3.7 Some Topology 


The word topology has to do with a family of subsets of a given fixed set. In the hopefully 


familiar setting of R, R” or C the paradigm of open sets are the open intervals, or open balls. 
These are not all the open sets but any open set in the given space can be built up by unions of 
these basic objects. These open sets come from the norms on the space, given a fixed P in one of 
these spaces, the open ball with P as center is {Q | ||P — Q|| < €} where €is a positive number 
and Q is from the space. One of the things that is basic to a first course in analysis (dealing with 


R” or C and the norm topology) is the idea of compactness. The idea of sequence has already 
been discussed and the ideas related to them (e.g. subsequences, limit infimum, etc.) are 
important for the concept of compactness in this setting. So a set in one of these finite 
dimensional normed spaces is compact if whenever K is a subset of such a space and all the 
elements of K are bounded by one fixed number then there is a sequence, say X;, in K, which 
converges to a point of K. Another important aspect of this is when K is the (norm) closed unit 
ball B in the space. That is the closed unit ball in a finite dimensional normed space is compact. 
That is there is sequence from B which converges (the points get close to each other and to the 
limit point). 


In the infinite dimensional situation this fails for the examples above with the norm topology. 
For example consider P and the sequence e; = (0,0,..,0,1,0,0...) with all zeros except in the 
kth position where we have placed a one. These are in the closed unit ball but 


llex—e|[P =2 GFK. 


Hence no subsequence of these terms can ever be close, when close means less than V2. The 
critical point here is that the closed unit ball in an infinite dimensional normed space can never be 
compact (again we are thinking of the topology induced by the norm). 

This lack of compactness of the unit ball makes the behavior of the linear operators appearing 
in the next section somewhat difficult to handle in many situations. Although we will not get into 
many examples we wanted the reader to be aware that there are deep topological difficulties in 
the study of these operators in the infinite dimensional normed spaces. 


2 _______________________~: 
3.8 Functions of Linear Operators 


In this section I will do my best to sketch some properties of bounded linear operators acting on 
Hilbert spaces (incidentally just to avoid any bad situations I am always thinking of spaces which 
are separable). There is an old facetious saying concerning what one reads in print “Believe what 
you read for it is true!”. I am going to apply this below in a few places (for short BWYRFUT). 

We see that it is possible to add linear operators and multiply by scalars and still remain 
within the realm of (bounded) linear operators. We also have the concept of multiplication played 
by composition. This operation is not commutative. For example on FP set 
T5(Z) = (22, 23, 24,..-) the backward shift. Then T5°T2(Z) = Z but T2°T5(Z) is not the 
identity. However one way to obtain more interesting examples of such operators is by taking 
nice functions f and considering the formal expression f(T) when T is a bounded linear operator. 
We want this to be another bounded linear operator (on the same space as T). For a simple 
example if f(z) = 2? + (4+ 22) and T is a bounded linear operator what is the operator f(T’)? 
Of course we want the z term to give us T? = T°T. What about the constant term? We think of 
powers of T (under composition) as above and T° = I the identity operator. So in this case f(T) 
is the operator 


f(T) =7T7?+(44+2)I=S. 


Now what about another “nice” function like f(z) = exp(z). We will write 6(X) to 
represent all the bounded operators on X. Note we could consider bounded linear operators from 
X into another normed space Y but considering X and Y as the same will make our point (and 
B(X, Y) as all the bounded operators from X into Y). The more general case is pretty much the 
same. 

At this point recall our definition of the word “complete”. We invoke our BWYRFIIT 
assumption and assert the fact that G(X) is complete and we ask the reader to accept that. That is 
if T|,1s a Cauchy sequence of operators (in the operator norm) then there is an operator T with T,, 


converging to T (again in the operator norm ||T;,, — T;,|| — 0,n, k — co implies there is a unique 


operator T with ||T,, — T|| — 0). Some of these ideas will appear as we continue with this 
example so we will not develop them any further. 
So let us consider the partial sums of f and then of f(T). 


! 
nan n=N 
Now ||T'|| = C is a fixed real (non-negative) number in this inequality and it is known that the 
series 
[e,¢) 
ae” 
ca nl 
M 
converges for any fixed C. So the partial sums above Ss; T "SH are the partial sums of a 
n. 
n=N 


convergent series and so are a Cauchy sequence. This implies the terms 7 as ” are Cauchy 
sequence in 8(X) and so by completeness of B(X) they converge to an pension S € B(X). If 
we had looked at 3 —T "(x) with « € X we would get an idea of how to prove the statement 
about the Somipleness of B(X). 


At this juncture we need some even more technical material on Hilbert spaces and operators 
on them. First assume that H;,7 = 1,2 are Hilbert spaces. We wish to make a sum of these 
spaces (which we can do as vector spaces) into another Hilbert space. Namely, writing 


H = H, @ Hp = {(z, y) : © © Hy,y © He} 


this is a vector space with pointwise addition and scalar multiplication. That is 


(x,y) + (u,v) i (x+u,y+v), 


and for a number c we have c(z,y) = (cx, cy). So that handles the vector space part of the 
definition; now for the inner product part we define 


< (x,y), (u,v) >= <2,u>n, + <Yy, 0 >HL; 


where the inner products are in the appropriate spaces (and we will drop the subscript notation 
when it is obvious which space we are working with). This yields for the norm 


< (x,y), (@,y) >= |lall? + |lyll?, 


again the norms are in the appropriate spaces. Further using the idea of sesquilinear forms (this is 
a use of BWYRFIIT) one can prove the following. For T a bounded operator on HH into H, there 


is another operator S from HH, into H, for which the following holds 
Tt. yo SYS 
This operator S is called the adjoint operator to T and is customarily written as T’’. 


Definition 9 An operator T on a Hilbert space H is called unitary if TT° = T°T =I (the 
identity). 


As an example recall our operator T3(Z) = (22, 21, 23,---, 2;,--) on U?. If we write a basis for 
P as E, = (0,..,0,1,0,0,..), where 1 is in the jth position and all other positions have a zero 
then it is easy to check that 


T,(E1) = EF, T,(E2) = Ey F Ti (E£;) = Ej,j > 2. 


Further, using the definition we have 


* 


T, (Fi) = Eo, T, (Eo) = £1, T, (Ej) = £;,9 > 2, 


* * 
so that one can check T, T; = T,T, = I, so Tis a unitary operator (and its inverse is its adjoint 
* 
T, ). 


Definition 10 An operator T on a Hilbert space H into a Hilbert space M is called a contraction if 
\|T'(ax)|| < ||a|| for all € X. That is the norm ||T|| < 1, and is a strict contraction if ||T|| < 1. 


Note that if T is any operator in 6(H, M) then the operator S = T’/||T'| is a contraction. 


Dr 


3.9 An Operator Equation 


Recall the form of an f € L, f(Z) = ages = W. Also recall that there are many different ways 


to express this rational function. Some of the most obvious are as follows 


f(Z) = (aZ+b)(cZ +d) * = (cZ +d) *(aZ +b) =a' + 4, 

as well as many others. So how do we lift this expression to an operator theoretic setting? If we 
just assume A,B,C,D and T are operators on various spaces and try to plug these operators into 
one of these expressions we have some immediate problems. Namely, since operators do not 
commute the second equality gives pause for thought. Which should we choose? Next the 
denominator will be an operator and we do not have (as yet) a way of thinking about such 
expressions. One of the ways to get a useful recipe to overcome and deal with these problems is 
to choose the following operator-theoretic form 


B— AT(I + CT) "D, 


where the letters A,B,C,D and T represent bounded linear operators on various Hilbert 
spaces.The importance of this choice is related to preserving contraction operators (see Theorem 
7). Note that if we return to the scalar form of this expression we have 

b(1+ceZ)—adZ _ (be—ad)Z+b 


1+cZ 1+cZ 


Now recalling the definition of the determining number of the linear fractional map in this form 
(see page 1) we require (bc — ad) — bc £0. This makes sense in that if ad = 0 then in the 
operator setting this would mean AD = 0, and this would imply our selected operator would be 
the constant operator B. So let us return to the Hilbert space H = H © Hp» discussed above. If T 
is in GH then we observe the following 


T((a,0)) = (u,v) = (A(x), C(a)) € H, 


where we claim the letters A and C represent bounded linear operators. A is a bounded operator 
in 6(H1), and C is in 6(H;, Hz). Let us check this claim. Choose #;, 7 = 1,2 in Hy. Consider 
the expression 


T((x1,0) + (x2,0)) = T((x1 + £2),0)) = (A(zi + 22), C(x1 + 22)) 


and using the linearity of T we deduce 


T((a1,0)) + T((#2,0)) = ((A(#1), C(a1)) + (A(22, 0), C(#2)). 


This of course implies that A and C are linear on their respective spaces. Similarly one shows 
that A(mx) = mA(x),C(max) = mC(zx) for scalars m. Now performing a similar argument 
with the expression 


T((0,y) = (u,v) = (Bly); D(y)); 


it can be shown that B € 6(H2, Hy), and C € B(Hz2). 


The discussion above shows that we may express an operator T on Has a matricial equation. 
In this expression we identify a point (x, y) € Has a “vector” 


() 


and we identify T with the “two by two” operator matrix 


(63) 


With this identification we have as a result 


‘an 


T((a,y)) = (A(z) + Biy), C(z) + D(y)). 


With this definition of T we define an operator expression 


M(X;T) = B- AX(I+CX)"*D=B- AxQ, 


‘am 


where Q = (I+ CX) "D, and X € 6(H2,H;), and C € 6(H1,Hz), and I is the identity on 
2. Thus (I + CX \ when it exists is a bounded operator on /, into itself, and so the operator 


‘am 


expression M maps HH, into H. One more piece of notation. Let 


Mr = {X © B(He,Hi) : 1+ CX is invertible in Hp}. 


A specific examples for which this is true is when ||C|| < 1 and X is a contraction. Motivation 
for this comes from the scalar case of the formula 


which converges when |t| < 1. Hence, if we write 


CO 
1 n n 
Trox =) (-)"(CX) 

oO 
this series will converge in the operator norm and this then provides the operator-theoretic 
meaning of the fraction on the left above. 

Now we come to our last BWYRFIIT statement but before stating it we have need of another 

definition. 


Definition 11 An operator T on a Hilbert space H is said to be positive if (Tx,x) > 0, all 
xe A. 


So for example if T is a contraction then we see that 
< (I-T'T)z,x >= ||a||? — ||T2||? > 0 


and so the operator (I — TT) iS a positive operator. The next statement is a robust lemma 
requiring several steps of computation and you might be able to muddle through it or you can 
check reference /3/, Chapter 7. The notation is that as above. 


Theorem 6 Assume T is a unitary operator on H. Then, for each X in Mr we have 


(I — (M(X;T)) M(X;T)) = Q"(I— X"X)Q. 
With this result in hand we culminate this section with the following theorem 


Theorem 7. Assume X € Mr. If X is a contraction, then M(X;T) is a contraction. 


Proof: We have noted that if X is a contraction then the operator (I — X"X) is a positive 
operator. Now consider the expression 


Q’(I— X*X)Q. 


We claim that this is also a positive operator. To check this assume x € Hy 
<Q*(I-X"X)Qz,2£>= Qx ?-— X(Qz) ?>0. 


But by Theorem 6 this implies that (J — (M(X: t) )M(X : T)) is a contraction and thus 
M(X : T) is a contraction. 


Now where are we in terms of taking this path into operator theory? Well at this stage we are 
at the brink of lots of interesting and deep operator theory and we are stopping here. One can take 
“square roots” of positive operators (that is if A is a positive operator there is another operator B 
with B? = A, and using these operators it is possible to make operator valued isometries for the 
setting of Theorem 7. If you have found any of this material interesting and want to pursue more 
of it I strongly suggest the references Fabian et. al. [2]. There is a great deal of material dealing 
with positive operators, the Wold decomposition and it is possible of course to get into this type 
of material for Banach spaces. 
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4.1 Introduction 


This is a short overview for the beginners (graduate students and advanced 
undergraduates) on some aspects of biholomorphic maps. 

A mathematical theory identifies objects that are “similar in every detail” from 
the point of view of the corresponding theory. For instance, in algebra, we use the 
notion of isomorphism of algebraic objects (groups, rings, vector spaces, etc.), 
while in topology homeomorphic topological spaces are considered similar in 
every detail. The idea is to classify the objects under consideration. 

In complex analysis the corresponding notion is the biholomorphism of the 
main objects in the theory: complex manifolds. Two complex manifolds M,, M, 
are biholomorphic if there is a bijective holomorphic map Ff’: My, — Mg. Like in 
those other theories one wants to find the biholomorphic classification of complex 
manifolds. This attempt is certainly an important motivation to study 
biholomorphic transformations. The classification problem appears to be a very 
difficult problem, and only partial results (though very important and highly 
interesting) are known. 

The Riemann Mapping Theorem states that any two proper simply connected 
domains in the complex plane are conformally equivalent. This is an exception. 


The general case is: for any n > 1 in C” any two “randomly” picked domains are 
not biholomorphic. We will mention three examples to support this viewpoint. We 
also are pointing out that in case n =1 the classification problem is well 
researched and mostly understood. For n > 2 it is way more complicated, and by 
now the pursuit of it has produced many very interesting and deep results and also 
created useful tools for SCV. Our intention is to put together various related 
problems, reflecting the authors interests, and in most cases refer the reader to 
some published material where details can be found. There are many surveys on 
most topics we consider. Quick search on MathSciNet shows that there are over 
two thousand papers published on biholomorphic maps. By no means this article 
is a comprehensive review. It should be considered as a glance into some of the 
results in the theory of these transformations. 

The approximate outline of this exposition follows. 

We start with three examples of non-equivalence for some “similar” domains. 
Then we introduce invariant metrics. Later we mention some important 
biholomorphic invariants and the related question of the extension of a 
biholomorphic mapping to the boundary. After that we consider the automorphism 
group of a domain in C”, which is a biholomorphic invariant, and related results. 
In the end we introduce and analyze “approximate” biholomorphic relations. 

So we start with some examples. 


ee 
4.2 Three Examples 


The Riemann Mapping Theorem guaranties the biholomorphism of two proper 
simply connected domains in C. But once we look at non-simply connected 
domains the situation drastically changes. Here's a result for annul. 

I). (F.H. Schottky, 1877) [52] Two annuli in 
C: A(r;, Ri) = {ri < |z| < Rj};7 = 1,2 are conformally equivalent if and only 
ifr, /Ry = r2/ Ro. 
Proof. (sketch) Denote the disc of radius r with center at the origin by A(r). 
Without any loss of generality we may assume that rj = r2 = 1, R, > Ro, and 
that f : Q(1, Ri) > Q(1, Re) is continuous to the boundary (we’ll address the 
necessity of this in a later section), and maps the unit circle OA(1) of 0(1, R1) 
into the unit circle of 0DA(1) C OM(1, R2). We will use the Schwarz reflection 
principle. The reflection over OA(1) extends f to an analytic map from 
Q(Rz', Ry) to Q(Rz*, Ra). We now continue to reflect over the smaller circle 
OA(R_') and get f extended to 2(R,*, R1). Continue this process indefinitely 
we get f extended to a conformal map of two punctured discs (0, R1) to 
(0,R2). Since f is bounded it can be extended to {0}; 
f : 2(0, Ri)  Q(0, Rz). Since f is now holomorphic, near the origin it will 
satisfy |f(z)| < C|z| for some constant C. This means that for large n we have 
|R,”| < C|R,"|. This cannot happen since by assumption R; > Ro. Oo 

For n > 1 in C”’, a theorem analogous to the Riemann Mapping Theorem 
does not hold even in the following case. 

II). (Henri _—_— Poincaré, 1907) For n2>2 the unit ball 


BY = £1 Fie 2) Elz)? <1 and the polydisc 
& 


A” = {(21, .--; Zn) ||Zi| < 1,7 = 1,..., 2} are not biholomorphically equivalent. 

Later on several proofs of this theorem will be provided. 

In C’, n > 1 small perturbations of the unit ball may create a continuum of 
non-biholomorphic domains. 

II). (Burns-Shnider-Wells thm, 1978) We’ll present one consequence of the 
main theorem from [7]. 

Consider the unit ball B® C C", n > 2,€ > 0. Then in the €— neighborhood 
of B” there is an uncountable number of mutually non-biholomorphic domains 
with real analytic boundaries, containing the ball and homeomorphic to the ball. 


Eee 


4.3 Some Invariant Metrics 


By introducing an invariant metric into a domain in C” one can then consider 
every biholomorphic map from one domain to another as an isometry with all the 
consequences coming out of that. There are several invariant measures: 
Kobayashi, Caratheodory, Bergman, and others. We refer the reader to a number 
of books written on this. It is quite an extensive topic, and we will only touch on 
this subject here. It will end with a proof of Poincare's example mentioned in the 
previous section. 


Definition 4.3.1. Let D be a domain in C”. The Carathéodory and Kobayashi 
infinitesimal pseudo-metrics are functions from D x C” to [0, 00) defined by 


Cp(z,v) =sup{|dg(z)(v)| : 9 € OD, A), g(z) = 9}, 
Kp(z,v) =inf{|u|:ueC, f € G(A,D), f(0) = z,df(0)(u) = v}. 


The Kobayashi indicatrix of D at z is 
Ing = {v € C": Kp(z,v) < 1}. 


Kobayashi Extremal maps exist when D is bounded. 

It was proved by Royden ([48] pp. 125-137). that every Kobayashi hyperbolic 
complex manifold is infinitesimally Kobayashi non-degenerate. The converse is 
false ({37] Remark 3.5.11). 


Proposition 4.3.2. Cp < Kp. 

Proof. Let z € D and v € C”. Let € > 0 be given. There is an f € O(D, A) and 
u € C” such that f(0) = z, df(0)(u) = v, and |u| < Kp(z,v) +e. There is a 
g € O(D,A) such that g(z) = 0, and |dg(z)(v)| > Co(z,v) — €. By Schwarz 
lemma, |d(g° f)(0)(u)| < |u|. It follows that 


Cp(2,v) — € < |dg(z)(v)| = |d(g’ f)(0)(u)| < [ul < Kn(z,0) +. 
Letting « — 0 yields that Cp(z,v) < Kp(z, v). Oo 


Theorem 4.3.3. The Kobayashi infinitesimal pseudo-metric is decreasing under 
holomorphic maps in the sense that if f¢€@O(D,92) _ then 


Koa(f(z), 4f(z)(v)) < Kp(z,»). 


Proof. Let ¢ >0. Choose he G(A,D) and wEC so that h(0) =z, 
dh(0)(u) =v, and Kp(z,v)+e>|ul. Since f°he O(A,2), and 
d(f°h)(0)(u) = df(z)(v), we see that |u| > Ko(f(z),df(z)(v)). Thus 
Kp(z,v) +e > Ko(f(z), df(z)(v)) for each e > 0. Oo 


Corollary 4.3.4. The Kobayashi infinitesimal pseudo-metric is invariant under 
biholomorphic maps in the sense that if f is a biholomorphic mapping from D 
onto 2 then 


Ka(f(z), df(2)(v)) = Kp(z, 2). 
Moreover df (z)(Ip,z) = La,p(z)- 


Theorem 4.3.5. Letn > 1. Then A” and B" are not holomorphically equivalent. 

Proof. Suppose that gis a biholomorphic mapping from B” onto A”. Let a = g(0) 
. There is an h € Aut(A”) with h(a) = 0. Let f =h’g. Then f(0) =0. By 
Corollary 4.3.4, df(0)(B”) = A”. That is impossible because B” has a smooth 
boundary and A” does not. Oo 


ee 
4.4 Boundary Extension of a Biholomorphic Map 


A set of biholomorphic invariants has been introduced by S. S. Chern, J.K. Moser 
[12], and N. Tanaka [53]. These invariants can be used for C‘~ manifolds of co- 


dimension one in C”. To use them for the case of a biholomorphism f : D; + D» 


of bounded domains in C” with smooth boundaries one needs first to prove that f 
can be extended smoothly to the boundary. We will now discuss this question of 


extending a biholomorphic map between two bounded domains in C” to a 
diffeomorphism or even a homeomorphism between the closures of these 
domains. 

In the case of one complex variable the conformal map f: D— 92 for 
bounded domains in C can be extended to a diffeomorphism D—> 2 if the 
boundaries 0D, 092 are smooth, and to a homeomorphism if these boundaries are 
piece-wise smooth simple closed curves. An examination of this problem in C one 
can find in [46]. 

In the case of C”,n > 2 the situation is much more complicated. A short 
counterexample to the extendability is given by Fridman [20]: two domains D, 2 


are constructed in C, both biholomorphic to the bidisk, both have piece-wise 
smooth boundaries, and there is a C'™ diffeomorphism between them that extends 
smoothly to their closures. However, no biholomorphic mapping F' : D — 2, nor 
F~'! can be extended continuously to the boundary. If D and Q are strictly 


pseudoconvex domains in €” with C’'® smooth boundaries the result of N. 
Vormoor (and independently G. Henkin) [35, 58] shows that any biholomorphic 
map between them extends to a homeomorphism of their closures. 

The question of smooth extendability is a very difficult problem. A major 
breakthrough was made by C. Fefferman in 1974. He proved the following 
theorem [14]. 


Theorem 4.4.1. Let f be a biholomorphic map from a domain D C C” to another 
domain (2c C”. Suppose that D and Q are bounded strongly pseudoconvex 
domains with C'~-smooth boundaries. Then f extends C'® to the boundary. 


Fefferman's original proof consists of two parts. First he considers the 
behavior of the Bergman kernel near the boundary, which leads to precise 
estimates of the Bergman metric near the boundary (for definitions and main 
results regarding the Bergman kernel see the book by S. Krantz [41]). In the 
second part, he proves the theorem using those estimates and the above mentioned 
Vormoor's result. We would like to note that [14] is 65 pages long, and the 
introduction (8 pages), has a very clear sketch of both parts of the proof. 

Following this major accomplishment, there have been a number of 
publications refining and presenting different proofs of the theorem. The book 
[41] has a clear and different proof of this theorem. We should also mention two 
more proofs that are widely cited: those by F. Forstneric [15] and S. Bell & E. 
Ligocka [4]. 

The proof of F. Forstneric is based on two classical results: the edge-of-the- 
wedge theorem and the Julia-Caratheodory theorem (for these classical results, 
see e.g. [49, 50]). In this sense it is considered elementary. 

Bell and Ligocka's proof of Fefferman's theorem made use of the Bergman 
kernel. For a bounded domain D C C”, the Bergman kernel function K(z, w) is 


defined by K(z, w) -» y ;(z)y;(w), where {y;} is an orthonormal basis for 


the Bergman space B. := L?(D) 1 H(D). The Bergman kernel does not depend 
on the orthonormal basis and it has the reproducing property that for each 


feB (D )= | Kew) fw) w) dw. 


Let Wj denote the closure of C§°(D) in the Sobolev space W*(D). Let 
H*(D):=W*(D)N AH(D) be the subspace of W*(D) consisting of 
holomorphic functions. A domain D C C” is said to satisfy Condition R if it is 
bounded with smooth boundary and if for every s > 0 there exist constants 
M>0O and C>0 such that the Bergman projection P: L?(D) > B?(D) 
satisfies ||Pf||zs(p) < Cll fll ws): 


Theorem 4.4.2. (Bell and Ligocka [4]) If D, and D2 satisfy Condition R, then 
any biholomorphic mapping between D, and Dz extends smoothly to the 
boundary. 


It had been known that subelliptic estimates for the O-Neumann operator 


imply Condition R and that the 0-Neumann operator satisfies the subelliptic 
estimates for bounded strictly pseudoconvex domains with smooth boundary. 
Thus Bell and Ligocka's theorem implies the Fefferman Theorem. 

Bell and Ligocka firstly proved that Condition R implies the following two 
conditions: 

Condition A. K(-,w) € C®(D) for each w € D, 

Condition B. For each zp € OD, there are n + 1 points ag,...,@,, in D such 


that K(z9, a9) 4 0 and 
K (Zo, a;) 
det 0. 
(is (20, @;) } i=1,....n * 


j=0,1,...,n 


Using Conditions A and B, they proved that for a given biholomorphic map h 
between domains D, and D, the Jacobian functions Jh(z) and (Jh(z))~* are 


bounded on D,. Assume that (Jh(z))~' is not bounded. Then there exists a 


sequence 2, — zo € OD, such that Jh(z,) — 0. The transformation rule for the 
Bergman kernel function 


Ki (Zn, a) = Ko(h(zn), h(a)) Jh(zn) Sh(a) 
and Condition A yield that Ky(zo,a) = 0 for each a € Dj, which contradicts 
Condition B. 
Consider a point z) € OD, and a sequence {z,,} in D, with z, — Zo such that 


h(z,) tends to a point s9 € OD. By Condition B, there are points bo, bi,...,5n 
such that K'(s9, bo) # 0 and 


K2(s0, b;) 
Q :— det OK» = 0. 
i=1,...,n 


a5~ (80; 95) 
j=0,1,...,n 
Let v;(s) = Ko(s, b;)/Ka(s, bo), eee Then 
det (34 (so)) = K2(s0,bo) ” "Q £0. Set a; = h-*(b;) and 
u;(z) = Ki(z,a;)/Ki(z,ao),j =1,...,n. The relation 
Ou; Ov; Jh(a;) Oh; 
(T= (zn) = (F= (sn) =)» (S= (en) 


Jh(ao) Oz; 


then tells us that si (z,) are bounded. This implies that si are bounded on D,. 
Therefore, his continuous up to the boundary. In particular h(zo) = so. Since 


det ( $2 (z0)) # 0, and det ( 52 (so) ) # 0, 


the functions u, and v; are smooth local coordinates near 2) and so respectively. 
With respect to these coordinates the mapping / is expressed as a linear mapping 


Jh(a;) 
Th(ao) 
boundary in a neighborhood of Zp. 


given by vj; = u;. It follows that h can be extended smoothly to the 


ee 


4.5 Symmetry: Automorphism Group of a Domain 


4.5.1 Automophism groups 


A biholomorphic map of a manifold M onto itself is called an automorphism of 
M. The set Aut() of all automorphisms of M forms a group. This group is 
clearly a biholomorphic invariant. There is a relatively recent survey of this group 
by S. Krantz [40], and we’ll refer to this paper throughout the section. 

A good example of using Aut(D) as invariant is Poincare's original proof of 
biholomorphic non-equivalence of the unit ball B” and polydisk A” (for n > 1) 
by comparison of their automorphisms groups; they happen to be Lie groups of 
different dimensions and therefore not isomorphic. 

There has been a lot of effort devoted to the study of Aut(D). As with all 
known invarians, this one is interesting (giving an idea of how “symmetric” a 


domain is) but by no means defining (even for n = 1 two annuli have the same 
automorphism group but might not be conformally equivalent). Moreover, after 


many deep studies one may conclude that for a general “random” domain in C” 
the Aut(D) = {id}. However, there is a large set of domains with non-trivial 
group, and this is a justification for detailed study of Aut(D). 

One of the first results was the paper by H. Cartan (1935) [10] which proved 


that Aut(D) is a real Lie group for any bounded domain Din C”. Bedford-Dadok 
and Saerens-Zame [3, 51], proved that every compact Lie group can be realized as 
an Aut(D) for some strictly pseudoconvex domain with a smooth boundary in a 


suitable C”. 


4.5.2 Description of automorphism groups and characterization of domains, 
manifolds with a given group 


Below for convenience we will use two notations for the holomorphic 
automorphism of D: Autp or Aut(D). 


Let D be a complex manifold. Consider a fixed point w¢ D. The 
automorphisms of D which fix w form the isotropy group at w, 


Autpw := {yp € Autp: p(w) = wv}. 
For y € Autp the left coset of Autp,~ with respect to g is defined to be 
pAutpw:={yg:9€ Autpy}. 


For y,~eAutp, if ply < Autp, then pAutp, Ny~Autp, =O; if 
w ty € Autpy then pAutp,» = pAutp,w. The cosets space 


Aut p/Autpy := {pAutp» :y € Autp} 


is naturally a real analytic manifold. The map from the cosets space 
Autp/Autpw to the orbit Autp(w):={g(w):qeAutp} given by 
pAutpw > y(w) is a bijective, real analytic map. Thus we have the following 


Proposition 4.5.1. The orbit Aut p(w) is a real analytic submanifold of D. Its 
dimension is dim Aut p(w) = dim Aut p — dim Aut pw. 


Let D be a bounded domain in C” or a Kobayashi hyperbolic manifold of 
complex dimension an. It follows from the normal family theory that Autp~ is 


compact. If ¢ € Autp, is an automorphism which fixes w, its tangent map dpy 
is a member of GL(n,C). When dy, = id, by looking at iterations of series 
expansions of g at w, we see that g must be the identity map. This implies that the 
map Autp.w — GL(n,C), p> dyy, is injective. Thus Aut p,» is isomorphic to 
a compact subgroup of GL(n,C). Since each compact subgroup of GL(n, C) is 
conjugate to a subgroup of the unitary group U(n), it follows that Autp is 
isomorphic to a subgroup of U(n). Thus dim Autp,» < dim U(n) = n?. 

A few examples of the Aut(D): 

1. The automorphism group of the projective space: Aut(P”) is isomorphic to 
PGL(n +1, C). The proof can be found in [1], p. 41. 

2. The detailed construction of the automorphism group of the ball B” and the 
polydisc A” can be found in [1, p. 54]. As it is shown in that book the Aut(B”) is 
a real Lie group of dimension n? + 2n, and dim(Aut(A”) = 3n. So, these 
groups have different dimensions for n > 1, and therefore not isomorphic for 
t= 2. 

Here's a way to describe the Aut(B”). 

Let 0 < A < land let f : B” — C” be defined by 


X 
filz) = ase 
f;(z) = viE® 2, ce eres 


(4.1) 


Then f € Aut(B”) and f(0) = (A,0,...,0). Fix a point a 0 in B”. There is a 
unitary transformation U such that Ua = (|a|,0,...,0). Let h be the 
transformation in (4.1) with 2 replaced by |a| and let g=U 1‘°h°U. Then 
g € Aut(B”), g(0) = a, and g(—a) = 0. The transformation g can be expressed 
as 


= Fr + Gas Qa(2), 


9(z) — dea '  l4+za 


(4.2) 


where 


It follows that every point in B” is mapped to the origin by some member of 
Aut(B”), hence the Aut(B”) action is transitive and B” is a homogeneous 
domain. Thus dim Aut g».9 = 2n. Since each element of the unitary group U(n) 
is an automorphism of B” that fixes the origin, we see that U(n) is naturally a 
subgroup of Aut pn 0. Thus dim Aut p.9 = n?. Therefore 
dim Aut(B”) = n? + 2n. 

The group U(n, 1) is the group of linear transformations under which the form 
X,X,+---+X,X,, — XoXo is invariant. So 


U(n,1) = |W € GE(n + 1,0): wT ° )W=($ a 


In homogeneous coordinates, B” is given by 


X,X1 ee +X,Xn =iXo Xo < 0. 
A b\. 
Each W = a) U(n, 1) gives amap gw € Aut(B”), where 
c 


Az+b 
gw(z) < a . 


(4.3) 


The map gy is the identity map if and only if W € S' := {e"I:t € R}. Thus 
PU(n,1) :=U(n,1)/S! is considered a subgroup of Aut(B”). Since 
(n+ 1)? —1=dim PU(n,1) < dim Aut(B") =2n+n2, we see _ that 


Aut(B") = PU(n,1). 
av1 +t? :) 
0 vi 
d=av1+t?, where V € U(n — 1), |a| = 1. Take a = 1, V = I,_; to obtain 
h (ag 0 
0 Les 
VS) V1+#2. The map in (4.3) is the transformation given by (4.2) with 
a= d~'b. 
3. The automorphism group of the entire space CC”, n > 2 is not finite 


If b=(t,0,...,0), t>0, c=b", then A=( 


) and d= /1+t?. Then gy is given by (4.1) with 


dimentional. Say, for © it contains shears (w, = 21; we = 22 + f(z1)) for any 


entire function f. 
4. More examples one can find in A. Isaev's book [36]. 
5. One more interesting result. It can be proved that if Dis a bounded domain 


in @ and dim(Aut(D)) = n? + 2n, then D is biholomorphic to B". A. Isaev 
gave explicit classification of all domains D in C’ for which 
n? —1< dim(Aut(D) < n? + 2n. These results are fully presented in [36]. 


4.5.3. Greene-Krantz conjecture 


In case of strictly pseudoconvex domains with smooth boundary only the unit ball 
has a NON-compact group of holomorphic automorphisms. This was proved by 
B. Wong [59]. The non-compactness of Aut(D) means that for at least one point 
zo € D the orbit Aut(D)(zo) has an accumulation point p on the boundary. J. P. 
Rosay [47] extended Wong's theorem by proving that if p is a strictly 
pseudoconvex point of OD then D is biholomorphic to the ball. The proof of 
Wong-Rosey theorem can be done by the scaling method ((40], p. 227). In the 
same survey one can find various versions of this theorem (even for infinite 
dimensional cases). 

Because of these theorems, the description of smoothly bounded domains with 
non-compact automorphism group becomes interesting. All the known examples 
of such domains reveal that the accumulation point on 0D is of “finite type in the 
sense of Kohn, D’ Angelo, and Catlin”. (A boundary point of a domain in C is of 
finite type if the boundary has finite order of contact with complex manifolds 
through the point; precise definition for C” can be found in [40]). In 1991, R. 
Greene and S. Krantz made the following conjecture [30]: 

Conjecture. Let D be a smoothly bounded domain in C”. Suppose that x € D 
has a boundary orbit accumulation point for the automorphism group in the sense 
that there are automorphisms ¢,; € Aut(D) and a point p © OD such that 
d;(z) — pas j — oo. Then pis a point of finite type. 

There have been many attempts to resolve this conjecture, only partial results 
have been obtained by now; for a more detailed discussion on it see [40]. 


4.5.4 Narasimhan's question 


In this section, we will again use the notation Autp instead of Aut(D). 

A bounded domain 2 C C” is said to have Property N if there exists a 
compact subset K of 0 with the property that for each z€ (2 there is an 
f € Autg such that f(z) ¢ K. For z€ 9 and a subgroup T of Auta, 


T(z) := {f(z): f © I} is the T-orbit of z. If S C 2, then I'(S) denotes the 
union of the I’-orbits of the points in S: 


IP(S) := Uzes F(z) = {f(z): 2€ S,f € I}. 


Thus, Property N is equivalently defined as follows: 0 is said to have property N 
if there isa K CC 2 such that Autg(K) = 2. The following is a question by R. 
Narasimhan: 


Question 1. If 0 is a bounded domain in C” with Property N, is 0 necessarily a 
homogeneous domain? 

A discrete subgroup of Auty is a subgroup of Auty that is discrete in the 
compact open topology, the default topology of Aut. Recall that we say a 
subgroup [ of Aut, acts on O freely if each f € I’ is fixed-point free. 

The following theorem is by S. Frankel [16]. 


Theorem 4.5.2. Let Q be a convex hyperbolic domain in C” and suppose that 
there is a discrete subgroup I. C Autg such that 2Q/I is compact. Then Q is 
biholomorphic to a bounded symmetric domain. 


Frankel first proved a distortion theorem for convex holomorphic embeddings 
and used this to reduce a complex analysis problem to one in affine geometry. 
Then he applied rescaling to produce a continuous family of automorphisms. His 
particular technique of boundary localization was very different from what had 
gone before, and he called it the rescale blow-up. 

Kazhdan conjectured that each irreducible bounded domain which admits both 
a compact quotient and a one-parameter group of holomorphic automorphisms 
must be biholomorphic to a bounded symmetric domain. Frankel [16] first 
confirmed the conjecture for bounded convex domains. Subsequent work by 
Nadel [44] and Frankel [17] proved it in general. 


Theorem 4.5.3. Let M be a compact complex manifold. Then the group Aut yy is 
a complex Lie transformation group and its Lie algebra consists of holomorphic 
vector fields on M. 


The above theorem is due to Bochner and Montgomery [5, 6]. For bounded 
domains in C”, we have the following theorem of H. Cartan [9, 11]. 


Theorem 4.5.4. Let D be a bounded domain in C”. Then the group Aut p is a Lie 
transformation group and the isotropy subgroup Autp,, at any point z € D is 


compact. If X is in the Lie algebra of Aut p, then JX is not in the Lie algebra of 
Aut D.- 


Theorem 4.5.5. (Kobayshi [39, p. 81]) Let M be a hyperbolic manifold. Then the 
group Aut y is a Lie transformation group and the isotropy subgroup Aut yz, of 
M at any point x € M is compact. 


The essential reason for the above three theorems is the following 


Theorem 4.5.6. (Bochner and Montgomery [6]) Let G be a locally compact group 
of differentiable transformations of a manifold M. Then G 1s a Lie transformation 


group. 


Theorem 4.5.5 is also based on the following early result of van Danzig and 
van der Waerden [13]. 


Theorem 4.5.7. Let M be a connected, locally compact metric space and Gyy the 
group of isometries of M. Then Gy, 1s locally compact with respect to the 
compact-open topology. 


SaaS 
4.6 Determining Sets and Fixed Points 


Let M be a complex manifold, f : M@— M a holomorphic map. z) € M is a 
fixed point for fif f(zo) = Zo. 

The following is a result in the classical function theory [43, 45]: if 
f : M — M isa conformal self-mapping of a plane domain M which fixes three 
distinct points then f(¢) = ¢. 

This one-dimensional result is true even for endomorphisms of a bounded 
domain D Cc C. To prove this one needs to first use the well known theorem, 
stating that if an endomorphism of D fixes two distinct points, then it is an 
automorphism; and then use the above cited [43, 45] theorem. 

We now introduce two notions to discuss how this result can be extended for 
higher dimensions. 

For a complex manifold M let H(M,M) be the set of holomorphic maps 
from M to M, 1.e., the set of endomorphisms of M. The group of holomorphic 
automorphisms of M, Aut(M) is a subset of H(M, M). 


Definition 4.6.1. A set K Cc M is called a determining subset of M with respect 
to Aut(M) (H(M, M) resp.) if, whenever g is an automorphism (endomorphism 
resp.) such that g(k) = k Vk € K, then gis the identity map of M. 


So, any three points of a plane domain D form a determining set for Aut(D) 
as well as for H(D). 

The other notion is Fia(f), it denotes the set of fixed points {x € M | 
f(z) = a} off. 

So, if Mis a plane domain, and Fix(f) is discrete, then the cardinality of this 
set #Fix(f) <2 for any f according to our remarks at the beginning of this 
section. 

We now have two classes of problems to investigate. First: the description and 
properties of determining sets for various complex manifolds. There are many 
results in classical analysis and topology proving the non-emptiness of Fix(f), or 
finding it for a given function. Our second class of problems to discuss is different 
from those: for a given M which sets can be Fizx(f) for some holomorphic 
f © H(M,»M). 


4.6.1 Determining sets 


The notion of determining sets was first introduced in [25]. That paper was an 
attempt to find a higher dimensional analog of the above one-dimensional result. 
Determining sets (for automorphisms and endomorphisms) in case of bounded 
domains in C” have been further investigated in the following papers [26-29, 38, 
54, 55]. 

Let's first look now at some examples of discrete non-determining and 
determining sets (from [25]). 


Example 4.6.2. Let A= {z¢C:1/2 < |z| < 2}. This is an annulus in the 
plane. The map 7(z) = 1/z has two fixed points (i.e., 1 and —1), yet r is not the 
identity mapping, so these points are not a determining set. Since the Aut(A) is 
well known one can check that any two points in general position in A do form a 
determining set. 


Example 4.6.3. In €* consider a shear of the form r(z,w) = (z,w+ ¢(z)), 
where @ is any entire function on the plane. Then 7 is a biholomorphic map of C. 
If @ has infinitely many distinct zeros then ¢ will have infinitely many fixed 


points, even though 7 is not the identity. So, the set of these zeros does not form a 
determining set for Aut(C?). 
By contrast, any biholomorphic (conformal) map of C that fixes two points must 


be the identity. So, any two distinct points in C form a determining set for the 
Aut(C). 


Example 4.6.4. It can be shown that a biholomorphic map of the unit ball B’ in €” 
that fixes n + 1 points in general position (in the usual sense of topology) must be 
the identity. One may check this by using the description of the automorphism 
group of the ball given in the previous section. We leave the details to the 
interested reader. So, this set of n + 1 points forms a determining set for the ball. 
We also note that no set of n points in B” forms a determining set. Indeed, let 
P1,---;Pn be the n points. Since the ball is a homogeneous domain, we may 
consider g € Aut(B”) such that g(p1) = 0. Now the set (g(p1), g(p2) .--, 9(Pn)) 
lies in a linear space L of dimension dim(L) <n -—1. Therefore, there is a 
rotation f € Aut(B”) that keeps all the points of L fixed. So, the automorphism 
h=g ‘fg € Aut(B”) is not an identity and fixes all the n points (pi, ..., Dn). 


Example 4.6.5. Consider the domain 
—— { (Z1, 22) eC. zu)" + zai < I any integer m => 2. Then any 
automorphism of U,,, that fixes two points in general position must be the identity. 
This result follows because the automorphism group of U,, is well-known to 
consist only of rotations in each variable separately. 

Contrast this example with the result from the last example (for the unit ball in 


c”). 


Example 4.6.6. Let U,,, be one of the domains from the last example. Let V be 


any rigid domain in C” (here rigid means that the domain has no automorphisms 
except the identity). Then, for an arbitrary chosen pair of points z € U;,w € Um, 
and an arbitrary x € V, any automorphism of U,,, x V which fixes both (z, x) and 
(w,x) will be the identity. For instance, the points z= ((1/2,0),xz) and 
w = ((0,1/2), x) will do. 

We will present now a few results when the determining set K is discrete and 
refer the interested reader to the above mentioned papers for more results and 
unsolved problems. 

We will need two more notions. Let M be a complex manifold. Let W,(M) 
denote the set of s-tuples (z1,...,25), where 2; € M, such that {71,...,2,}isa 


determining set with respect to Aut(M). Similarly, W,(M) denotes the set of s- 
tuples (a ,...,2,;) such that {x,,...,2,} is a determining set with respect to 


H(M,M). So W,(M) C W,(M) C M®. We now introduce two values s9(M) 
and §9(M). In case Aut(M) = id, s9(M) = 0, otherwise s9(M) is the least 
integer s, such that W,(M) #Q@. If W,(M) = 9 for all s then so(M) = oo. 


Analogously symbol 89(M) denotes the least integer s such that W,(M) + 0, if 


no such integer exists (i.e. W,(M) = 0 for all s) then 89(M) = oo. In all cases 
8o(M) — §9(M). 


1. For hyperbolic manifolds of dimension n the following estimate holds 
89(M ) <n-+l1. 

First we note that this estimate for bounded domains in C” was proved by J.P. 
Vigue (see [55]). In that paper a more precise theorem is proved. Let a be a point 
in a bounded domain D. Then there is an open set U C D” such that 


(a,...,a) € U and for all (21, ..., Zn) € U, (21,.--; Zn) € W,(M). 

For a general hyperbolic manifold M one may consider a small Kabayashi ball 
b € M such that it is biholomorphic to a domain D C C”. Pick a point a € M. 
Let f: M — WM be a holomorphic map such that f(a) = a. Consider a small 
Kobayashi ball b = b(a,€) whose closure is compact in M, and such that b is 
biholomorphic to a bounded domain DCC”; let h:b—D be such a 
biholomorphic map. Note that since the Kobayashi distance is non-increasing 
under holomorphic maps, we have f : b > b, and therefore g = hfh~!: D > D. 
By using the above mentioned result, one can pick n points 21,..., 2, € D, such 


that Z = (h(a), 21,-.-)2n) € Wn4i(D). Consider the set of n+1 points 
h-'(Z) = (a,h-+(z),...,k 1(2n)) Cb. If our function f € H(M,M) (in 
addition to a) is also fixing all points h~'(z;), i.e. f|n-1(z) = id, then g|z = id 
and therefore g = id. We conclude that f|, = id, and consequently f = id. So, 
h-*(Z) € Wnii(M), and therefore 89(M) <n+1. 

2. The above statement implies same inequality for automorphisms of a 
hyperbolic manifold M, so(M) <n+1. 

However, for automorphisms much more information can be provided. s9(M) 
depends on how large the group Aut(M) is. If Mis a bounded domain in C” ( 
dim(M) =n) then the general estimate (s9(M) <n+1) can be refined to 
so(M) < n for domains that are not biholomorphic to the unit ball B” C C”, and 


the only hyperbolic manifolds for which s9(M) = n+ 1 are those biholomorphic 
to the ball. This gives a characterization of the ball in C”. 

3. If a positive integer s > s9(M), then W,(M) 4 Q, so there are s points 
such that if an automorphism of M fixes these points it will fix any point of M. 
Now the question arises whether the choice of these s points is generic. The 
answer is positive for any hyperbolic manifold M: W,(M) C M* is open and 
dense if not empty. Similar topological properties for the determining sets of 
endomorphisms of a general hyperbolic manifold do not hold. 


4.6.2 Isolated fixed point sets, cardinality 


In classical mechanics the following Euler's theorem is well known: the general 
displacement of a rigid body with one point fixed 1s a rotation about some axis. 


So, if one considers an orientation-preserving isometry of a domain in R° fixing 
one point, the fixed point set of this isometry will necessarily contain at least a 


segment, so the fixed point set cannot be a discrete set. In the euclidean space R”, 
one can always find a domain which has a euclidean isometry with exactly one 


fixed point, however for any n, if an isometry of a domain in RK” has two fixed 
points it will force the existence of at least a segment to belong to the fixed point 
set, and so this set will be at least one dimensional. 

Switching to complex analysis, we remark that any holomorphic 


automorphism of a bounded domain in C” (or in general, hyperbolic manifold) is 
an isometry in an invariant metric, so an Euler type statement is certainly 
meaningful, that is if this automorphism has a discrete fixed point set one can 
inquire what its cardinality and structure might be. To describe this more 
precisely, let f : M — M be a holomorphic self-map of a complex manifold M. 
Suppose that Fix(f) is discrete. We shall examine mostly two questions. First, 


how large this set can be for specific cases: M is a bounded domain in C”, a 
hyperbolic manifold, etc., while f is a holomorphic automorphism or 
endomorphism. Second, the structure of Fix(f), namely which points of M could 
form such a set for some holomorphic self-map of M. Everywhere below we 
consider only holomorphic self-maps (automorphisms or endomorphisms) of 
various complex manifolds, and for the sake of compactness the word 
holomorphic may be omitted. 

In examining the cardinality of a discrete fixed point set, let's first consider the 
situation in one dimension. For a bounded domain D C C the discrete fixed point 
set of a holomorphic map f : D — D can have no more than two points. This 
follows from the above mentioned observation: any set of three points in D must 


be a determining set for endomorphisms. An annulus gives an example of a 
domain that has an automorphism with exactly two fixed points. 


In C” the situation is not yet completely clear. Here are a few statements we 
know. 

1. For a convex domain one has the following theorem: the isolated fixed point 
set of any endomorphism consists of at most one point. This statement follows 
from the main theorem in [56]: such a set has to be connected. The proof is based 
on establishing that in Bergman metric there is a unique geodesic connecting two 
fixed points, and it (the geodesic) will then also belong to the fixed point set. 

2. For a bounded strictly pseudoconvex domain D in C” with real analytic 
boundary the number of points in a discrete fixed point set of an automorphism is 
finite. Moreover, there is a number m = m(D) such that #(Fix(f)) < m. 

Here's a proof of this statement. If D is biholomorphic to the ball or if n = 1, 
then the statement is clear. Assume that n > 2 and D is not biholomorphic to the 
ball. By a theorem in [57], there is a neighborhood U, of D such that each 


automorphism of D extends to be an injective holomorphic map on U,. Consider a 


g € Aut(D). Choose domains U2,U3 with smooth boundaries so that 
D cc U3 CC U2 CC Uj. For every h € Aut(D) in some neighborhood of g, 
h(OU2) is so close to G(OU2) that h(OU2) NM g(U3) = 0. Since h(U2) is a 
connected component of C”\h(OU2) and since h(U2) > D, we see that 
h(U2) D g(U3) for every h € Aut(D) in some neighborhood of g. Since Aut(D) 
is compact, there is a neighborhood Q of D such that Q C g(U;) for each 
g€ Aut(D). Let U be the interior of the intersection of the sets 
g(U1), g € Aut(D). Then U D Q and g(U) = U for each g € Aut(D). That is, 
each automorphism of D is also an automorphism of U. There is a finite cover of 
open sets {Vj: 7 =1,...,m} of D such that each pair of points in a Vj is 
connected by a unique distance-minimizing geodesic with respect to the Bergman 
metric of U. Let f € Aut(D). If f fixes two points in a Vj, f must fix each point 
on the unique distance-minimizing geodesic connecting the two points. 
Consequently, each Vj contains at most one isolated fixed point of f Therefore, 
the number of isolated fixed points of f is < m. 

3. Must the cardinality of an isolated fixed point set of an automorphism or 
endomorphism be bounded by a number depending only on the dimension of the 


manifold under consideration? For endomorphisms of bounded domains in C” the 
answer is negative. It is also negative for automorphisms of a general hyperbolic 


manifold and the entire €”. However, for an automorphism of a bounded domain 


in C” the answer is not yet clear. Let's consider several examples demonstrating 
some of the results. 


Example 4.6.7. For any k € N, there exists a bounded domain D C C", n > 2, 
and a holomorphic endomorphism f : D — D, such that #-(Fiz(f)) = k. 

Proof. Without any loss of generality we can present an example for n = 2. 
Let S be the open Riemann surface in 
C?: $= {(z,y) € C?|y? = (x — ay)...(a — ax) }, where ay, ...,a% are k distinct 
points in C. The restriction g of (x, y) — (x, —y) to S has exactly k fixed points. 
Following ((34], VIII, C8, p. 257) there exists a holomorphic retraction 
p:V-—S of an open neighborhood V of S onto S. Now the mapping 
f:=9 p:V-V has exactly k fixed points. Of course Vis not bounded, but we 
can consider a bounded open set W C V,(a;,0) € W for all s = 1,...,k and 
such that g(W) = W. This bounded domain will have the same property. 


Example 4.6.8. There exists a hyperbolic manifold with a holomorphic 
automorphism whose fixed point set is discrete and consists of an infinite number 
of points. 

Proof. Consider the submanifold X of D* defined by y*? = B(x), where D is 
the open unit disc and B is a Blaschke product with an infinite number of zeroes, 
the restriction to X of the map (xz, y) > (x, —y) is an automorphism of X and has 
an infinite number of isolated fixed points. 


Example 4.6.9. For any n>2 and any k€ N, there exists a polynomial 
automorphism f of CC”, such that #(Fix(f))=k. Moreover, let 
n > 231, p2,---) Pk are k distinct points in C". Then there exists a polynomial 
automorphism g € Aut(C”) such that Fixz(g) = {p1, po, .--,Pk}- 

Proof. Let a,,...,a, be k distinct complex numbers. Consider the map 
H:C” — C” given by 

Wy = 21 + 224 (21 ay)(Z1 — a2)...(Z4 — ar) 

Wz = Z2+ (21 — ay) (z1 — a2)...(Z4 — ar) 

w, = 12, for alls = 3,..,7 

This map is an automorphism, whose fixed point set is the set of the following 
k points: (a1, 0, ...,0), (a2, 0, ..., 0), ....., (ax, 0, ..., 0). 

Now p; = (a;,b;),a; € C,b; € C”'. Without any loss of generality we 
assume that the a’ s are all distinct (in case they are not, one can first use an 


invertible linear transformation of C” to achieve this). Consider the polynomial 
transformation F: w; = z1,w' = z'+ f(z1), where f:C—+C"! is the 


Lagrange interpolation polynomial map satisfying f(a;) = b;, w’ = (wa,..., Wn). 
Then F(a;,0) =pj,j =1,...,k, and F € Aut(C”). Now the automorphism 
g — F’H° F 1 is such that Fix(g) = {p1, po, .-, pe}- 


Some problems 

1. Let D be a bounded domain in C", f € Aut(D), and Fix(f) is a discrete 
set. Can #(Fix(f)) = 00? 

If one considers the domain D C C” which is a direct product of n annuli, one 
can then find an f € Aut(D) with #(Fix(f)) = 2n. So, the next natural 
question is 


2. Let n > 2,D be a bounded domain in C”, with a piecewise smooth 
boundary, f € Aut(D), and Fiaz(f) is a set of isolated points. Can 
#+(Fix(f)) > 2n+1 ? (As noted earlier, for n = 1 the answer is negative). A 
more restricted version of this question is 

3. Is there a number m such that for any strongly pseudoconvex domain 
DccC", ODE C®, and f € Aut(D), if Fix(f) is a set of isolated points, 
then #:(Fiz(f)) < m, where m = m(n) (i.e. m depends on the dimension only)? 


4.6.3 Fixed point sets consisting of one or two points 


In this section we’ll discuss which subsets of a manifold D can be Fix(f) for 
some holomorphic automorphism or endomorphism f:D— D. As the title 
suggests we’ll consider two cases. 

First we consider the case when every single point of a domain is the Fiz(f) 
for a suitable holomorphic f 


Theorem 4.6.10. /28, Theorem 2.1] If every point of a hyperbolic manifold D is a 
fixed point set for some holomorphic automorphism of D, then D is a 
homogeneous manifold. 


Proof. 1. First we note that the theorem will follow from a local statement: let 
x € D, then there exists a neighborhood U, of x such that for any y € U, there is 
a g © Aut(D) such that g(y) = «. Indeed, if this is true consider two arbitrary 
points a, b € D, connect them by a compact path L, cover L by a finite number of 
U,,z € L, and one can obtain an f € Aut(D), such that f(a) = b. 

2. We now prove the local statement. Let x € D. By [29], for each point 
x € D there is an invariant Hermition metric in some neighborhood of the orbit 
G(x), where G = Aut(D). Consider a small enough ball b(z,€) in that metric 
with center x and radius €, € > 0 will be determined by the construction later. Let 


y € W(x, €); consider the orbit O(y) ={z¢€ D: dg © Aut(D), 9(y) = 2}. 
Consider now a point p € O(y), such that d(x, p) = d(x, O(y)), where d(-,-) 
denotes the distance function induced by the local invariant metric. Clearly, 
p € O(a, €). If p = z, there is nothing to prove; otherwise consider a small ball b; 
of radius <d(z,p) that lies inside O6(x,d(z,p)), and such _ that 
0b; M Ob(z, d(x, p)) = p. This construction is possible if €is small enough, fixing 
such an € = €(x), we denote b(z, €) = Uz. 

We observe that O(y) N b(z, d(x, p)) = @. Let gq denote the center of the ball 
b,. By the assumption of the theorem there exists an h € Aut(D) whose fixed 
point set is g. Now h(p) # p, and h(p) € Obj, since h(0b,) = 0b;. We now 
conclude that h(p) € O(y)Mb(z,d(z,p)), which contradicts the previous 
observation that this intersection is empty. Therefore z = p € O(y), and the 
theorem has been proved. Oo 

We now provide the following example. 


Theorem 4.6.11. There exists a domain D in C with infinite number of points each 
of which ts the fixed point set for a holomorphic automorphism of D. 


Proof. Consider D = C\ Ue A(n, 1/3) where A(n, 1/3) is a disk with center at 
NE 


n€Z and radius 1/3. Consider f,: z— (—z+(2k+1)). Then for any 
keZ,f, € Aut(D), and its fixed point set consists of one point 
Fiz(f) = {k + 1/2}. Oo 

Let's now consider pairs of points as fixed point sets. Though such domains 
exist, no domain can have too many pairs of distinct points as a fixed point set for 
an automorphism. 


Theorem 4.6.12. Let DCCC”. The set N C D? of all pairs, each of which 
cannot be a fixed point set for a holomorphic automorphism of D, contains a full 
measure set in D?. 


It follows from the following two Lemmas. The first is a classical statement 
(see [8] p. 80; also [54] thm 2.3) proving that for z € D and its isotropy subgroup 
I,={f(z) =z, f € Aut(D)} there is a local system of coordinates where each 
f € I, isa linear map. 


Lemma 4.6.13. There exists a holomorphic map ¢:D-—>(C” such that 
o(z) = 0, ¢'(z) = id, and for all f € I, one has ¢° f = f'(z)°¢. 


The theorem will now follow from the second lemma. 


Lemma 4.6.14. Let D CC C",a € D. Then there exists a complex analytic set 
ZC D,(dimZ <n), such that ifb € D\Z then the two points {a,b} are such 
that for any automorphism f fixing these two points, the fixed point set of f is at 
least one (complex) dimensional. 


Proof. Using the previous lemma we first find the function @ for the point a. Let 
Z = {z € D\d(z) = 0}. If b € D\Z, then suppose f € Aut(D) and f fixes both 
points a and b. We have f’(a)-¢(b) = o(f(b)) = o(b). Since by choice 
o(b) £0, and @ is biholomorphic in the neighborhood U of a, for a number 
A, |A| > 0, small enough, there exists a point c€ U C D,c a, d(c) = AG(b), 
and f(c) € U. Now 6(f(c)) = f'(a) - O(c) = f"(a) - AP(B) = AG(B) = GLC). 


Since @ is biholomorphic in U, we have f(c) = c. Oo 


i 


4.7 Approximate Biholomorphisms 


4.7.1 ‘Approximate’ biholomorphism: exhaustion 


As noted in the introduction two randomly picked domains in C” are likely to be 
non-equivalent (=non-biholomorphic). Can they be “approximately” equivalent? 
Let's make this question precise. Consider a sequence of bounded domains 
{Di}~, CC”, such that all D;, C D and jim Di = D in some topology of 


domains in €”. So, D can be approximated by D, for large k. Suppose now that 
there is a domain G CC C” such that each D, is a biholomorphic image of G, 
fx: G — Dx a biholomorphic map onto D,. In this case we will say that D can be 
exhausted by G. So, Gis approximately equivalent to D. 

So, if given a bounded domain G in C” what can it exhaust? The list A(G) of 
these domains is of course a biholomorphic invariant. The range varies widely 
depending on G, and we’ll mention several examples. By definition A(G) 
contains G. We will show that if Gis homogeneous then A(G) consists only of G 
itself: 


Theorem 4.7.1. If G,D are bounded domains in C", G is homogeneous and D can 
be exhausted by G (in the Hausdorff topology), then D is biholomorphic to G. 


By the way, since the ball and the polydisc in C” for n >1 are non- 
biholomorphic it follows that neither can exhaust the other and there should be 
“largest” imbedding of each into the other. The precise estimates have been 
obtained by H. Alexander [2], which can be considered as another proof of the 
Poincare's Example. 

If Gis a smooth, bounded strictly pseudoconvex domain, then A(G) has only 
the following elements: G and the unit ball B’[21]. Comparing this example with 
the above theorem shows that the notion of exhaustion is not symmetric: it 
happens that D € A(G) but A(D) does not contain G. 

For half-the ball in €’,n > 2, S = {z: z= (2,...,2n) € B”, Re(z,) > O}, 
A(S) contains the unit ball and the polydisk [19]. This, by the way shows that S is 
not biholomorphically equivalent to any smooth, bounded strictly pseudoconvex 
domain (though, of course, this can be proved by other means). 

The above theorem shows that a homogeneous domain exhausts only itself. 
On the other side of the spectrum is the following domain. 


Theorem 4.7.2. (Fridman [18]) There exists a universal domain U CC C” which 
can exhaust any other bounded domain in C” (in the Hausdorff topology). 


So, this domain is “almost” equivalent to any other domain in C”. Any 
domain/set that has the approximation property described in this theorem we will 
call a universal domain/set. 

As we noticed at the start of this exposition, there is no version of the 


Riemann mapping theorem in C”. The above statement can be considered the 


approximate Riemann mapping theorem for any C”. 

A short explicit construction one can find in [18], it will not be repeated here. 
We’ll make a few remarks concerning the universal domain. 

There is a great flexibility of constructing this domain. That is, such a domain 


can have the automorphism group isomorphic to Z,, for any k € N. Also one can 


construct many such non-biholomorphic domains; this gives an example of two 
domains that can exhaust each other but are non-equivalent. 

There is a great variety of universal domains. And most of them have one 
curious property we are about to describe. The exposition is short and elementary, 
but the statements are useful when dealing with general biholomorphic mappings 
and therefore we are including them here. It would be helpful for the interested 
reader to know the construction before reading about the following property of U. 
It is as follows. There is a designated point p € OU such that to approximate a 


domain G C C” with a given precision €, one has to find a small 6 and a 
biholomorphic imbedding T:U-—-G, such that the 06-neighborhood 
W; = B(p, 6) NU will “blow up” to cover most of G while T will squeeze 
everything else outside that neighborhood in U almost to a point. This will 
accomplish the goal: T(Ws) C G will be &close to G, while the rest of T(U\Ws) 
will be small enough to not create a larger approximation mistake. We use the 
notation B(p, 6) for a ball at center p and radius 0. 


So, by an obvious association we can call p a “source” of all domains in C”, 
and the described property a “Big Bang property”. One can also express this 
property by stating that for any 6 > 0 the set B(p,d)NU is also a universal 
domain. Is the existence of such a “source” necessary for a universal domain? We 
prove the following 


Theorem 4.7.3. Let U be a universal domain. If the boundary OU does not contain 
any complex analytic variety of dimension one then OU contains a “source”, that 
is such a point p € OU that B(p,6) QU is a universal set for any 6 > 0. 


Corollary 4.7.4. If n= 1 then any universal domain has a “source” on the 
boundary. 


The proof of the Theorem is based on a generalization for several complex 
variables of a one-dimensional Hurvitz theorem. For completeness, we include the 
proof. 


Lemma 4.7.5. Let G,D be bounded domains in C”. Suppose that there is a 
sequence of domains {V;,}, Vi, C G, and maps Fj, : Vk, — D such that 

1. F,(V;,.) = D and F, is a biholomorphic mapping. 

2. For any compact K C G 

a) there exists a number m such that V, > K,s > m, and 

b) the sequence {F,} for s > m tends uniformly on K toa map F: K > C”. 

If F(G) contains a point z9 € D then F is a biholomorphism between G and 
D. 


Proof. 1 Evidently F(G) c D. We want to show that F(G) C D. Let wo € G be 
such a point that F'(wo) = zo and € > 0 be so small that the balls in Kobayashi's 
metric By = Bp(z0,€) CC D and By = Be(wo, 2€) CC G. Let z € By for a 
large enough k, F;,(wo) € Bi. Therefore, EF (@) € By. Let w be the limit point 
of {F,'(z)}. Evidently, F(w) = z. We have proved that F(G) > By. Fis a limit 


of regular holomorphic mappings. Since Gis connected Fcan be either regular at 
every point in Gor the Jacobian of F vanishes on G. In the latter case F(G) could 
not contain any open set (by Sard's theorem). Since F(G) D Bi, Fis regular on 
G. This implies that Fis an open mapping, so F(G) C D. 

2. We will show now that Fis one-to-one. Let w’, w” € G. For a large number 
k and the Kobayashi metric p we have 


pa(w',w") = pe(Fy'° Fr(w’), Fy °° Fe(w")) 
< po(Fi.(w'), Fe(w")) 
< po(Fr(w'), F(w')) + pp(F(w'), F(w")) + po(F(w"), Fe(w")). 


When k-—oco we obtain pg(w’,w”)) < pp(F(w’), F(w”)). Hence, if 
F(w’) = F(w”") then w’ = w”. 

3. To finish the proof we have to show now that F(G) D D. Without any loss 
of generality we may assume, passing to a subsoquence if necessary, that {F,"} 
converges uniformly on compacta to f : D > G. Repeating the first step of the 
proof for this mapping we obtain f(D) C G. For the mapping F’’ f : D > D and 
any z © D we have F” f(z) = jim [F,° F, '(z)] =z. 

Hence, F(G) > D completing the proof of the Lemma. Oo 


Proof of the Theorem 4.7.3: In [18], the existence of two universal 
holomorphically non-equivalent domains is proved. Let G be one of them that is 


not biholomorphically equivalent to U.G can be represented as G = Ve where 


open sets Vi C Vz, CC G for all k. Since U is universal there exists a sequence 
of biholomorphic imbeddings f;, : U — G such that f,(U) D Vz. Consider now 
U;, = fx(U) and F;, = f, '. Since {F,} is a sequence of bounded holomorphic 
maps we may assume, taking a subsequence if necessary, that {F;,} converges 
uniformly on any compact K C G. Let F = lim Fy, F: G — U. Since Uis not 
equivalent to G, F : G + OU according to the lemma. Since OU does not contain 
any analytic curves, F(G) = pis a point on OU. Let 6 > 0. We are going to prove 
that U' = B(p, 6) NU is a universal set. Let Mbe any domain in C”, K a compact 
in M. Since G is universal, there exists a biholomorphic imbedding g: G > M 
such that g(G) > K. Denote K; = g-'(K) —compact in G. {F;,} converges 
uniformly on K, to p. Therefore, there exists a number N such that Fiy(i,) C U' 
. Consider now fy = Fy’ :U' > Gandh=g’ fn : U' > M. According to the 
construction h(U’) > K. This completes the proof of the Theorem. Oo 


Using the same Lemma, we now prove the Theorem 4.7.1. 
Proof. Pick two points: zo € G, wo € D. Since G is homogeneous then we may 
assume that for all k, fx(zo) = wo. Now, one can see that the sequence {f;,} 
converges uniformly on compacta and its limit is f : G — D a biholomorphism.p 


4.7.2 Upper semicontinuity of automorphism groups 


It is a general geometric observation that “small perturbations can destroy 
symmetry but not create symmetry”. In case of domains in C€”, one may interpret 
this statement more precisely. Let {D;,}7°., be bounded domains in C” and this 


sequence tends to a bounded domain D Cc C” in some topology on domains in C”. 
Loosely put, is it possible that Aut(D;) is “larger” than Aut(D) for all k large 
enough? If that was possible, perturbation (in the given topology) of D of any 
small size can create domains with more symmetry. This question is referred to as 
non-semi-continuity property for automorphism groups. 

We will mention here several statements and examples pertaining to the 
question; for more detailed discussion on semi-continuity and open questions see 
[22—24, 40]. 

In the early eighties, R. Greene and S.G. Krantz ({31—33]) examined this 
question. They proved an upper semicontinuity result in the C’ topology, and gave 
the first counterexample to the upper semicontinuity principle. In [23, 24, 42] the 
semicontinuity question has been examined further for various other topologies. 
We also note here that in Riemannian geometry results of this kind have been 
obtained by various authors. 

Here's the statement in the C'‘® topology: the semi-continuity holds in this 
case; for a more precise statement see [40, p. 232]. 


Theorem 4.7.6. (Green, Krantz) Let U C C” be a smoothly bounded, strongly 
pseudoconvex domain. Let U' be another smoothly bounded domain whose 
boundary is sufficiently close to OU in the C® topology. Then Aut(U') is a 
subgroup of Aut(U). 


In [32], the same authors provide the following counterexample of the failure 
of upper semicontinuity in the C'~£ topology (for any € € (0, 1)). 


Example 4.7.7. There are pseudoconvex domains {D jhjet and Dp, each of which 
is C™ and strongly pseudoconvex except at one point, such that Aut(D,;) 4 {id} 


for all 7 > 1, Aut(Do) = {id}, and D;-—+ Do in the C+ topology, any 
€ € (0,1). 


The other statements and examples below are in the topology induced by the 
Hausdorff metric; we denote the corresponding metric space by H”, space of all 


bounded domains in €” with the metric equal to the Hausdorff distance between 
boundaries of domains. 


Example 4.7.8. There is a bounded C’‘® domain D in C that is not simply 
connected, and such that for any neighborhood U of D in the Hausdorff metric 
there is a C® domain D in U such that Aut(D) is not isomorphic to any 
subgroup of Aut(D). 


Construction. B(zo,r) denotes an open disk with center z) and radius r. 
Ny ae 
Consider an (IV + 1)-connected domain D = A\ U, A, where A = B(0,1) is 


the open unit disk and all A, are smaller non-intersecting closed disks, whose 
boundaries lie entirely in A. For a given 1 > € > 0, fix a positive €; < € and such 
that the set S = {z € A|Re(z) > —1+€,} contains all A. Suppose a natural 
number 7 >1 is also given. We now choose a positive 6 such _ that 
L(S) Cc B(1,1/27) where L is a Mobius transformation L(z) = 4**, and 


1+2za’ 
a=1—-6. We observe that L(A) =A. Consider now 


1 
M= 7) (L(D) exp(#%i) (each term is a rotation of L(D) by angle ars One 


can verify that by construction Z; acts on M. We define D; = L~*(M). Then 
D; C D, and since Aut(D;) is isomorphic to Aut(M), Z; is isomorphic to a 
subgroup of Aut(D;). Also the difference D\D,;C A\S' and therefore the 


Hausdorff distance between D and D,; is less than €. If N > 2 the group Aut(D) 
is finite. Since 7 could be chosen to be arbitrarily large, the statement has been 
proved. 

Remarks. The above construction works for any finitely connected domain: in 
any neighborhood of this domain and any integer j there is a domain whose 


automorphism group contains Z;. A similar construction can be done in C” for any 
m 21, 


Theorem 4.7.9. (Fridman and Poletsky [23]) Let M be any domain in C”. Then 
there exists an increasing sequence of bounded domains M; C Mx, CC M 
such that M = UM,, and Aut(M;,) contains a subgroup isomorphic to Zx. 


This statement can be proved by using the remark in the first sub-section: for 
each n € N there exists a universal domain whose automorphism group has a 
subgroup isomorphic to Z,. 

So, for any domain (even a rigid one, i.e. with Aut(D) = {id}) one can make 
a perturbation of less than a given size and obtain a domain with a large cyclic 
group. The natural question arises: which Lie groups will a similar statement hold 
for? It will hold for any finite group: 


Theorem 4.7.10. Let G be a group of order m < oo. For any n > m the set of 
bounded domains in C" whose automorphism group contains a subgroup 
isomorphic to G is everywhere dense in H". 


A detailed proof of this theorem is given in [22]. 

So arbitrarily small perturbation of a domain in C” may create a domain with a 
larger automorphism group. But in provided examples the groups are discrete, of 
dimension zero. The natural question arises: can small perturbation in H” create 
domains with larger dimensions of automorphism groups? The following answer 
is “no”. 


Theorem 4.7.11. (Fridman, Ma, Poletsky [24]) The function dim(Aut(D)) is 
upper semicontinuous on H". 
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5.1 Introduction 


Since the foundational work of Kohn on subelliptic theory of the 0-Neumann Laplacian on smoothly bounded 
strongly pseudoconvex domains ([Ko063, Ko64]) and that of Hérmander on the L?-estimates of the 0-operator on 
bounded pseudoconvex domains in C” ((H65]), it has been known that, unlike the classical Dirichlet or Neumann 


Laplacian, existence and regularity of the 0-Neumann Laplacian closely depend on geometry of the underlying 
domains (see the surveys [BSt99, Ch99, DK99, FSO1] and the monographs [CS99, St09]). It is natural to expect 


that spectral behavior of the 0-Neumann Laplacian is also more sensitive to geometric properties of the underlying 
domains than the classical Laplacians. 

In spectral analysis of self-adjoint differential operators, two problems stand out: discreteness and positivity of 
their spectra. Both are widely studied problems in physical sciences with important ramifications. Discrete spectra 
are seen in many physical phenomena such as light emission and string vibration, while positivity is related to 
whether a physical system has positive ground state energy. 

Positivity of the classical Dirichlet Laplacian is well understood. As a consequence to the classical Hardy 


inequality, the bottom of the spectrum of the Dirichlet Laplacian on a domain in R” that satisfies the outer cone 
condition is positive if and only if the inradius of the domain is finite (cf. [D95]). Whereas positivity of the 


Dirichlet Laplacian is not sensitive to boundary geometry of the underlying domain, positivity of the 0-Neumann 
Laplacian, as we will see, is. 


In this Chapter, we study the 0-Neumann problem from spectral theoretic perspective. Our emphasis is on the 


interplay between spectral behavior of the 0-Neumann Laplacian and geometry of the underlying domains. This is 
evidently motivated by Marc Kac's famous question “Can one hear the shape of a drum?” ([Ka66]). Here we are 


interested in determining geometry of a domain in C’ from positivity of the 0-Neumann Laplacian. (See [Fu05, 
Fu08] for related results.) 

The plan of this paper is as follows: In Section 5.2, we review the classical setup of the 2-Neumann problem. 
In Section 5.3, we present a spectral theoretic setup for the 0-Neumann Laplacian and prove that these two setups 
are equivalent, and as a consequence, we establish the self-adjointness of the 0-Neumannh Laplacian. In Section 
5.4, we review the concept of pseudoconvexity. In Section 5.5, we provide a treatise of Hormander's L?-estimates 
of the Q-operator through the lens of spectral theory. In Section 5.6, we prove the converse to Hérmander's 
theorem. 

We have made an effort to present a treatment that is accessible and self-contained, modulus extensive usage of 


spectral theorems for self-adjoint operators. An excellent treatise on spectral theory of differential operators can be 
in Davies’ book ([D95]). 


ee 
5.2 The 0-Neumann Laplacian 


In this section, we review the classical setup of the 0-Neumann problem. In the subsequent section, we will present 
a spectral theoretic setup for the O-Neumann Laplacian and prove that these two setups are equivalent. As a 
consequence, we establish the self-adjointness of the 0-Neumannh Laplacian. 


5.2.1 The Cauchy-Riemann operator 


The 0-operator, also known as the Cauchy-Riemann operator, is arguably the most important differential operator 
in complex analysis. Let 


OA (a) ;_O o_ 1 (a) ;_O 
B=t(e-is) md F=3(8 +48). 
Let be a domain in C” and let f € C™(22). Define 


i n of 7 n of 
Of = >, Be 7 and Of = Be, ee 
j=1 J j=l J 


Thus Of :C~(2) > Coy (2) and Of : C°(2) — Co) (2). It is easy to check that the exterior differential 


n 


operator d= 0 + 0. Let yp = S> p,;dZ; © Doi(M), the space of (0, 1)-forms with smooth coefficients compactly 
j=l 
supported on 0. Then 


where 


is the formal adjoint of 0. Let Ley (2) be the space of (0,1)-forms with locally integrable coefficients. Let 


f € L°(2) andge€ Ley (2). We say that 


in the sense of distribution if 


(f,99) =(99), Ve © Dor(2). 
A locally integrable function f is holomorphic on Q if Of = 0 in the sense of distribution. Such a function is 


necessarily smooth. In fact, it has power series representation at any point in Q. 
Let 


Lio (2) = 1! = », fidz;|f; € 20} 
= 


be the space of (0,1)-forms with L?-coefficients, equipped with the standard Hermitian inner product with the 
corresponding norm given by 


fi\? dv. 


te. n 
isl > 


Note that here we use the convention that under the standard Hermitian metric on C”, (dz;, dz,) = 6;~ Where 
Oj = 1if j = kand by, = Oif 7 A k. 


We now extend the 0-operator to be a densely defined closed operator 


8: L*(Q) > Lia ,)(2) 


with 


Dom (8)={£ E L?(Q)| 3C > 0,|(£,8p)| < Cllyll, Ve € Fon(@)}. 
Namely, the domain of 0 is the space of all L?-functions f such that Of € Lio 1) (2) in the sense of distribution. 
The 0-operator acts on (p, q)-forms, 0 < p,q < n, is defined as follows. Let u € CR (2): Write 


Uy 
u= sa ) uzsdzy \ dzjz = ) u,zdzyz \ dZy. 
TJ Ts 


(5.1) 


where I = (i1,...,%p) is a p-tuple and J = (j1,...,j,) a q-tuple of integers in [1, n], dzy =dziA...\ %,, 
dz; = dz;, \...AdZj;,, and the prime indicates that the sum is taken over strictly increasing tuples. Then 


O43 CR (2) > CR ar (®) is given by 


U 
Ogu = 0 Ouzz Adzz A dzZy. 
TJ 


(We may suppress the subscript q when it is clear from the context.) Let Dpq)(§2) be the space of (p, q)-forms 
with smooth compactly supported coefficients. For ye Dopq) (2), the formal adjoint 


Bq: Depgqsi)(2) > Bepq)(2) of 0, is given by 


OurkK 


Ogu = (-1)?" S$ dz; \dZzx. 


k=1 T,K 07k 


where K runs over all strictly increasing q-tuples. 


Exercise 5.2.1 Show that 


(3,f,9) = (f, Bap); VfE CR (2), PE PDipq+1) (2). 


Let Lie) (2) be the space of (p,q)-forms with locally integrable coefficients. Let u€ Lie) (2) and 
ve Le 43) (2). We say that 
Oqu=v 
(5.2) 
in the sense of distribution if 
(2,9) = (uP), VP © Doar) (M).- 
(5.3) 


Let Li )(@) be the space of (p, q)-forms u with L’-coefficients, equipped with the standard Euclidean metric: 


U 


fell? = Sep Do Uleerall? = SO leer? 
TJ TJ 


for u given by (5.1). 


The 0-problem is to solve the 0-equation (5.2) and to study existence and regularity of the solutions. 


Definition 5.2.2 The operatord, : Li, ; 


(2) > Li 


(p,q+1) (2) is defined in the sense of distribution. More precisely, 


Dom (84)={u E Lig (2) | AC > 0,|(u, 9)| < Cllell, 
Vp € PD p,q+1) (2) } 


(5.4) 


and hence by the Riesz representation theorem, there exists v € Li, oi) such that 0,u = v in the sense of 
distribution as defined by (5.3). 


Lemma 5.2.3 (1) The operator 0, : Li 0 (2) > Li a (§2) is densely defined and closed. (2) 041° 0 


Proof: (1) Since Dipq)(@) C Dom (dq) € Lipg (@) and Dip.q) (2) is dense in Ling (2); so is Dom (84). Hence, 
ies is densely defined in Ling (2): Suppose that u; € Dom(0,) and wu; > u in Ling (2) and Ogu; > v in 
Lp qr1)(2)- Then for any y € Dpq+1)(2), 


(w xe) = Jim (u3,9¢ ) = lim (3,3, = (ne). 


Thus, u € Dom (Oq) and Oyu = v. Therefore, Og is closed. 
(2) This follows from 


—) 


dd = (Oq+1 T Bq+1)" (Og + 04) 
= Oy Og + Oq11° Og + Oq11°Oq + Og41 Og = 0 


and form degrees consideration. = 


Since here we will work only in C” with the flat Euclidean metric. There is no difference between the theory 
for (p, q)-forms and that for (0, g)-forms. For simplicity of notation, we will restrict ourselves to (0, q)-forms. 


Definition 5.2.4 The operatord, : Lig 41(2) > Lio) (2) is the adjoint of 04. Thus its domain is given by 


Dom (9,)={u EL gi(M) IC >0,|(u,5,v)| < Cllell 


Vu € Dom (8,)}. 
(5.5) 


As we will show shortly, u € Dom (3,) imposes a boundary condition on u when © has C!-smooth boundary 62 
and u € Co (2). 


REMARK 5.2.5 The O4-operator defined by Definition 5.2.2 can be considered as the adjoint of the formal adjoint 


Dg? Li, qt) (2) = Li, o(®) whose domain Dom (34)=Ypq+1)(2). It is sometimes referred as the maximal 


extension of Og : ipq)(2) + Hpq41)(2). The 5, -operator, the adjoint of 0, defined by Definition 5.2.4, is then 
the closure of Bo: It is sometimes referred as the minimal extension of Sy: 


5.2.2 Integration by parts 


Let Q be a bounded domain in R" with C'-boundary. Let r(x) be a defining function for Q, i.e. r € C1(R"), 
2 = {x € R"|r(x) < 0}, and dr A 0 on b92. The outward normal direction is then given byn= Vr/|Vr], By 


replacing r by r/|Vr|, we may assume that |Vr| = 1 on b§2. By the divergence theorem, we have for any 


a (Fi,. F ., Fy) E OC"), 
[v-Fae= | F-rds. 
2 b2 


(5.6) 

Let u,v € C1(Q). Applying (5.6) to Fwith F; = ud and F; = 0 for all k Ps 3, we have 

) / a 
(ud) dx = uv < dS. 
[, Ox; ay on Os 
Therefore, 
a ’ 
/ ode = -{ ugede+ [ ud ds. 
Q Ox; 7) ; bQ i 

(5.7) 


Now let 2 CC C” be a domain with C'-boundary. Let r = r(z) be a defining function of 0 such that |dr| = 1 on 
b22. 

The following formulas are complex versions of (5.7). Note that we will slightly abuse the notation and use 
(-, +) to denote both the pointwise inner product associated with the metric and the integral of the pointwise inner 
product over the domain. 


Lemma 5.2.6 Let u,v € C!(Q). Then 


(5.8) 
—10pcand 
(2,0) = (u,) +f wogeas. 
(5.9) 


To generalize the integration by parts formula to forms, it is convenient to study the contraction operator. For a 
k-form u and a vector €, the contractiongé su is a (k — 1)-form defined by 


€su(m1, re) Nk-1) = ul€, Migs »k-1)- 
(5.10) 
It is easy to check that 
E(u Av) = (és) Av + (-1)*8"w A (€.0). 
(5.11) 


Let I = (41,...,ip) and J = (j1,..., J). Let K be a tuple of positive integers. We will use J\K to denote the 
tuple obtained by deleting entries in K from J. We have 


Jy dx = O jk: 
Moreover, 
0, Leg J; 


(—1)* "denqn Adzj, l=is, 
—10pcand 


0, l J; 
(dz; A dZz) = pts— x. 
(1)? "der Ady, 1 = js. 


For a vector €, we define its dual form by 
e = ( é), 


where (-, -) denotes the pointwise standard Hermitian inner product on C”. Thus if 


O 
f= Cr si big): 


then 
(ajdz; + bjdz;) A 
j=l 


umm. 
I 


Similarly, we can define the dual vector of a 1-form. Let € be a (0,1)-vector, it follows from straightforward 
computation that for any (p, g)-form u and (p, q — 1)-form yv, 


(€.u, v) = (u,& A v). 


Given a (p, q)-form u and (p’, q')-form f with p’ > p, q’ > q, the interior productu V f is a (p' — p,q’ — q)-form 
defined by 


(uV f,g) =(f,uAg), 


for any (p’ — p,q’ — q)-form g. Therefore, we have 


fiu= & Vu. 
Using these notations, we have: 
n 
ya _ > @) 
0 = dz; x Oz? 
j=l 
n 
fee 
ga 02 


Lemma 5.2.7 Letu € Clog) (@)- Then 
(ude = (ou) +f (Br) 9) ds 
bQ 


Proof: This follows from simple computations: 


(5.12) 


As a consequence of Lemma 5.2.7, we have: 2 
Lemma 5.2.8 Let Q CC C” be a bounded domain with a C'\-smooth defining function r. Write 
i “. Or 4 


j=l 


Letu € Coo (2). Then u € Dom (0°) if and only ifT su = 0 on b2, which is 


Do Ua Ga, =O 
k=1 
(5.13) 
on b92 for every (q — 1)-tuple K. In this case, 0° u = Ou. 


Definition 5.2.9 For 1 < q <n — 1, the 0-Neumann Laplacian U, : Lio g\2) > Lio g(®) is given by 


—*¥ 


q— Oq 194-1 +0 


ce) 


* 
q-4@ 


with 


* 


(Q)|u € Dom (Bo) 0,10 € Dom (841), 


(0,9) 


u € Dom (0,), Ogu € Dom (3,)}. 


Dom (U,) ={uelt 


On the bottom degree forms, the 0-Neumann Laplacian is given by Dp = 55 Oo with 


Dom ( o)J={u € L?()|u € Dom (49), dou € Dom (85) } 


and on the top degree forms LU, = On id, 4 with 


Dom ( »)={u E Li, »)(@)|u € Dom (9,1), 9,11 € Dom (9,1). 


The 0-Neumann problem is to study existence and regularity of solution to the equation 


u = f. 


(5.14) 


REMARK 5.2.10 Suppose = {r(z) <0} be a bounded domains with C'-boundary. By Lemma 5.2.8, if 
u € C?(Q), then f € Dom (Op) if and only if 


on 622. This boundary condition resembles the Neumann boundary condition but they are not the same. For 
u = fdz \... A dZ, with f € C?(Q), u € Dom (O,) if and only if wu = 0 on b2, which is exactly the Dirichlet 
boundary condition. 


By Lemma 5.2.8, for a smooth form u in Dom (a), Ou=¥u. Therefore, on a smooth form that is in the 
domain of 0, we have 


=00°+0°0=89+90 


O 

_ > te) (a) (a) (a) > te) 

_ S dz, \ a (as 4be ) _— ) aE 43; (dz, A a ) 
jl j Jj 


Therefore, as a differential operator, the O-Neumann Laplacian in C” is just a constant multiple of the usually 
Laplacian, acting coefficient-wise on forms. (Indeed, as we remarked earlier, on top degree forms, it is one-fourth 


of the Dirichlet Laplacian.) What distinguishes it from the classical Dirichlet or Neumann Laplacian is the 0- 
Neumann boundary conditions: 


u€Dom(d") and du€Dom(d’°). 


(5.15) 


It does not follow readily from the above definition whether the O-Neumann Laplacian is densely defined or 


self-adjoint. This will be answered in the next section after we introduce a spectral theoretic setup for the 0- 
Neumann Laplacian. 


5.3 Spectral Theoretic Setup 


We now present the spectral theoretic setup for the 0-Neumann Laplacian. We will show that the 0-Neumann 
Laplacian defined by its associated quadratic form is consistent with the one defined through Definition 5.2.9. As a 


consequence, we establish the self-adjoint property of the 0-Neumann Laplacian and show that the domain of its 
square root is the same as that of its associated quadratic form. We will use #(T) and .V (T) to denote 
respectively the range and kernel of the operator T. 


Lemma 5.3.1. Let T : Hl, —> Hz be a densely defined and closed operator and let T” be its adjoint. Then the 
following are equivalent: 


1. &(T) is closed. 

2. There exists a positive constant C such that |\f|| < C\|T'f\| for all f € Dom(T) 0. (T)~. 

3. BT’) is closed. 

4. There exists a positive constant C such that || f\| < C\|T™ f|| for all f € Dom (T’) NW (Ty. 
Proof: We first prove the implication (1) > (2). In this case, T : Dom (T) NV (T)* > &(T) is a bijective 
closed map. Its inverse is also a closed map from the closed subspace # (T) into H,. Applying the closed graph 
theorem, we thus have (2). We now prove (2) = (1). Suppose Tf; — g. Write f; = f} + f? where f> € W (T) 
and f € W(T)~. Then by (2), ff is a Cauchy sequence in H,. Assume jim f; = f. Then Tf =g¢ &(T) 
since Tis closed. The proof of (3) < (4) follows the same lines as that of (1) <= (2). 

We proceed to prove (2) = (4). For any f € Dom (T) NV (T)* and g € Dom (T") A.W (T*), 


(a TF)| = (29 F)| SIT *gllllfll < CIT GIT SI. 


Since g € WY (T") =# (T), the above inequality implies that (4). The proof of the implication (4) = (2) is 
similar. 7 


Lemma 5.3.2 Let T : H; — Hb» be a densely defined closed operator. Let F be a closed subspace of Hy» such that 
FD &(T). Then &(T)=F if and only if there exists an positive constant C such that 


fl <CIT"f|, VF ¢Dom(T") NF. 


In this case, for any g € W (T)~, there exists f € Dom (T’) such that 0° f = g and || f|| < C\lgl|. 


Proof: We first prove the forward implication. It suffices to prove that 
B= {fe Dom(T*)nF||IT"f| <1} 
is a bounded set in H). Let f € Band let g € F’. Assume that g = Th for some h € Dom (T). Then 


KF.9)| = (Ff T)| = |(T°F, AD] S IIA < oo. 


Thus by the uniform boundedness theorem, the set B is bounded. 


We now prove the backward direction. Let g € F. Consider the linear functional T’* f ( f, 9) on & (2") To 
prove that this is well-defined, it suffices to show 


\(f.9)|< CIT" fli, Vf € Dom (T*). 


This obviously holds when f € F+ cC #(T)+=. (T*). When f € F, this follows from the Schwarz inequality 
and the assumption. By the Hahn-Banach theorem, there exists an h € Hl such that ( f, 9) = (T* f, h). Therefore, 
g=Th. Furthermore, since F! C .W(T"), we have #(T)=Z# (Tynpom(t*)), which is closed by the 
assumption. The last statement in the lemma then follow from Lemma 5.3.1. = 


Let Q be a non-negative, densely defined, and closed sesquilinear form on a complex Hilbert space H with 
domain Dom(Q). Then Q uniquely determines a non-negative and self-adjoint operator S$ such that 


Dom (S!”?)=Dom (Q) and 
Q(u, v) = (SV2u, SV/2y) 
(5.16) 


for all wu, v € Dom (Q) (see, for example, Theorem 4.42 in [D95]). 
For any subspace L C Dom (Q), let \(Z) = sup{Q(u, u)|u € ZL, ||u|| = 1}. For any positive integer /, let 


d;(Q) = inf{A(L)|L C Dom (Q), dim(L)=}}. 
(5.17) 


Since S is self-adjoint and non-negative, its spectrum o(S) is a non-empty closed subset of [0, 00). The bottom of 
the spectrum is given by inf o(S) = A1(Q). The essential spectruma,($) is a closed subset of o(S’) that consists 
of isolated eigenvalues of infinite multiplicity and accumulation points of the spectrum. Furthermore, o,(T) is 
empty if and only if A;(Q) — 00 as j + oo. In this case, \;(Q) is the /” eigenvalue of S, wherein the eigenvalues 
are arranged in increasing order and repeated according to multiplicity. The bottom of the essential spectrum 
inf o,(T) is the limit of X;(Q) as 7 —> co (see [D95, Chapter 4]). We set inf o.(S) = 00 if o-(S) = 0. 

Let Ty : Hy, — Hyyi, = 1,2, be densely defined and closed operators on Hilbert spaces. Assume that 


&(T1) C WV (T2), Let T. be the adjoint of T,. Then ve is also densely defined and closed. Let 
Q(u,v) = (T,u,T; v) + (Tou, Tov) 


with its domain given by Dom (Q)=Dom(T,) Dom (T3). The Proposition 5.3.3 shows that the 0-Neumann 
Laplacian as defined by Definition 5.2.9 is consistent with the one defined using the quadratic form as above. As 


the consequence, the 0-Neumann Laplacian defined by Definition 5.2.9 is self-adjoint (see Theorem 5.3.9 below). 


Proposition 5.3.3 Q(u,v) !!is a densely defined, closed, non-negative sesquilinear form. The associated self- 
adjoint operator 0 defined through (5.16) is identical to the one given by 


Dom (O)={f € Hz|f © Dom (Q), 
Taf € Dom (T;), Tif € Dom(T;)} 


(5.18) 


and 


=T, 4 Ts: 


Proof: The closedness of Q follows easily from that of T; and T,. The non-negativity follows from the definition. 


We now prove that Dom (Q) is dense in H). Since VW (T2)'=R (T,) CW (T,) and 
Dom (T2)=.V (T2) @ (Dom (T2) NV (T2)*), 
we have 


Dom(Q) = Dom (T;) M Dom (T2) 
= (NW (Tx) NDom(T;)) @ (Dom (T2) NV (T2)*). 


Since Dom (T;) and Dom (T») are dense in H,, Dom (Q) is dense in W (T2) ® W (T2)>=Hb. 
Recall that f € Dom (L) if and only if f € Dom (Q) and there exists a g € H» such that 


Q(u, f) = (u,g), forall u € Dom (Q) 


(5.19) 


(see, for example, Lemma 4.4.1 in [D95]). Thus 


Dom (L) > {f € Half € Dom(Q), Taf € Dom (T,), T;f € Dom (T,)}. 


We now prove the opposite containment. Suppose f €Dom(Q). For any u¢ Dom(T2), we write 


U= U1 + U2 € (YW ay Dom (T2)) 0% (i) Note that AY C R(T,) = (T2). It follows from 
(5.19) that 


(Tou, Tof)| = |(Tou1, Tof)| = |Q(ur, f)| = |(u1,9)| < |lull - lIgll- 


Hence Tf € Dom (T,). The proof of T, f © Dom (Ti) is similar. For any w € Dom(T;), we write 


w= wi+ we € (VW (T2) WDom (T;)) ® WV (T2)*. Note that W(T2)'=Z (T;) C N (T;). Therefore, by 
(5.19), 


(Ty w, Ty f)| = |(Ty wr, T, FY = |Q(wr, f)| = v1, 9)| < lel -llgll- 


Hence, it. f € Dom (T, )=Dom (Ti). It follows from the definition of 0 that for any f €¢ Dom(D) and 
u € Dom (Q), 


(Of, a) =(OV? fu) = OF, %) 
= (T; f,T,u) + (Tof, Tou) = ((TrT, +772) f,u). 


Hence, LJ = TT, + T, Ta. a 


Definition 5.3.4 Let Tbe a self-adjoint operator on a Hilbert space. (1) Tis positive if its spectrum o(L) C [e, 00) 
for some c > 0. (2) Tis essentially positive if there is a constant c > 0 such that o,(L) C |e, oo). (3) Tis gap 
positive if there is a constant c > 0 such that o(L1) N (0, c) = 0. 


Thus for a self-adjoint operator T, we have: 
positivity = essential positivity > gap positivity. 


The following proposition is due to Hérmander [H65, Theorems 1.1.2 and 1.1.4]. Let. YW (Q)=WV (T;) NN (T2) 
. Note that when it is non-trivial, WY (Q) is the eigenspace of the zero eigenvalue of o. When #(T;) is closed, 


NM (T2)=4(T1) © VY (Q). 


Proposition 5.3.5. The following statements are equivalent: 


1.0 is gap positive. 

2.0 has closed range & (CO). 
3. Both T, and T have closed ranges # (T,) and #(T>2). 
4. There exists a positive constant such that 


fI3< C?Q(,f), Vf €Dom(T;) Dom(T2), f L ¥ (Q). 


(5.20) 
Proof: Since # (T1) C WY (T2), 
H, =%W(T2)' @N (T2)= (T2)+ @ B(T1) O(N (T2) 0 (T1)*) 
= B (Tz) © B(T1) O(N (T2) NN (Ty)). 
The proposition then follows from Lemma 5.3.1. : 


The following proposition is well-known (compare [H65, Theorem 1.1.2 and Theorem 1.1.4], [C83, 
Proposition 3], and [Sh92, Proposition 2.3]). 


Proposition 5.3.6 The operator 0 is positive if and only if &@ (T1)=W (T2) and # (T2) is closed. 


Proof: Assume is positive. Then 0 is in the resolvent set of and hence has a bounded inverse 
G : Hy ~ Dom (LC). For any u € Ap, write u = T,T, Gu + T, T,Gu. Ifu ec WY (T2), 


0 = (Thu, TeGu) = (TT, T2Gu, T,Gu) = (T, T:Gu, T, TrGu). 


Hence, T, TrGu =0 and u= TT, Gu, Therefore, #(T1)=.V (T2). Similarly, R(T,)=N any. Therefore, 

T, and hence T, have closed range. To prove the opposite implication, for any wu € Dom(Q), we write 
1 

u =u + uy € WV (T2) @ VW (T2)*. Note that uy,u2 € Dom (Q). Since WY (T2)=# (T1)=N (T,) and T) 

has closed range, it follows from Lemma 5.3.1 that that there exists a positive constant c such that 


* 
clluuy||? < ||) wy ||? and cllug||? < ||T2u9||?. Thus 


¢lluel|? = ef |feea |]? + [lea]? < [174 wall? + [Touall? = Q(u,u). 


Hence, inf o(0) >c > 0. . 
For a subspace L C Hy, denote by Pz. the orthogonal projection onto L+ and T}|,. the restriction of T> to 


L+. The next proposition generalizes Proposition 5.3.6 (see [Fu05]). 


Proposition 5.3.7 The following statements are equivalent: 


1.0 is essentially positive. 
2.4 (11) and & (T2) are closed and W (Q) is finite dimensional. 


3. NW (Q) is finite dimensional, V (T2) 0 WY (Q)*=&(T1), and& (Ta|_y(qy:) is closed. 


4. There exists a _ finite dimensional subspace LCDom (T) AMN(T2) such that 
MN (Tz) NL+=P,1(# (T1)) and B (T9|,.) is closed. 


Proof: We first prove (1) implies (2). Suppose a = inf o,(L) > 0. If inf o(X) > 0, then -V (Q) is trivial and (2) 
follows from Proposition 5.3.6. Suppose inf o(L1) = 0. Then o(L) M (0, a) consists only of isolated points, all of 
which are eigenvalues of finite multiplicity of 0 (see [D95, Theorem 4.5.2]). Hence VY (Q), the eigenspace of the 
eigenvalue 0, is finite dimensional. Choose a sufficiently small c > 0 so that o(L1) M [0,c) = {0}. By the spectral 
theorem for self-adjoint operator, there exists a finite regular Borel measure w on o(X) x N and a unitary 
transformation U : Hy > L?(o(C) x N, dy) such that VOU! = M,, where M,(2,n) = xy(x,n) is the 
multiplication operator by x on L?(o(C1) x N, dy) (see [D95, Theorem 2.5.1]). Let Py(q) be the orthogonal 


projection onto WV (Q). For any f € Dom(Q) NW (Q)*, 
UPya)f =X0.Uf =9, 


where x 0,-) is the characteristic function of [0, c). Hence Ufis supported on [c, 00) x N. Therefore, 


Qf, f) = / _2|UF [du > ellUfl? = elf. 


o(Q)x 


It then follows from Proposition 5.3.5 that both T, and T, have closed range. 


To prove (2) implies (1), we use Proposition 5.3.5 in the opposite direction: There exists a positive constant c 
such that 


ell fl? < Q(f,f), forall f € Dom(Q)N.V(Q). 
(5.21) 


Proving by contradiction, we assume info,(11)=0. Let ¢ be any positive number less than c. Since 
Looe) = #(Xx(0,<)(O)) is infinite dimensional, there exists a non-zero g € Ljg,-) such that g | W (Q). However, 


Ug|? du < elUgll? = ellal?, 


Q(9,9) = i LX 0,¢) 


)xN 


contradicting (5.21). 
We do some preparations before proving the equivalence of (3) with (1) and (2). Let L be any finite 


dimensional subspace of Dom (T;) MN (T2). Let Hy = H2OL. Let T; = T2|yy, and let T? = T; |i. Then 


T; : Hy — Hs and T,! : Hl, —> H are densely defined, closed operators. Let Ty : H,; — Hj be the adjoint of T,!. 
It follows from the definitions that Dom (T;) C Dom(T‘). The finite dimensionality of L implies the opposite 


containment. Thus, Dom (T;)=Dom (T}). For any f € Dom (Tj) and g € Dom (T,')=Dom (T;) nt, 
as 9) = (f, T; ‘9) = Ch Ti 9) _ igs g)- 
Hence Tj = Pz °T, and @ (T})=Py1(#(T1)) CW (T). Let 
Q"(fi9) = (IVF, T1' 9) + (Tif, Trg) 


be the associated sesquilinear form on Hi, with Dom (Q’)=Dom (Q) N L+. 
We are now in position to prove that (2) implies (3). In this case, we take L = VW (Q) in the above settings. By 
Proposition 5.3.5, there exists a positive constant c such that 


AFA) =Q(F,f) 2 ellf|?, forall f € Dom (Q’). 


We then obtain (3) by applying Proposition 5.3.6 to Tj, Ty, and Q’(f, g). 
Finally, we prove (4) implies (1). By Proposition 5.3.6, we know that there exists a positive constant c such that 


Q(f, f) > ellf|?, for all f ¢ Dom (Q) NL*. 


The rest of the proof follows the same lines of the above proof of the implication (2) = (1), with WY (Q) there 
replaced by L. P 


Let 
H = WN (T2)/#(T1) 


be the quotient space of VW (T2) and # (T1). This is the cohomology of the complex 


T, T, 
Hy, > Hz 5 Hs. 


It measures the extent to which one cannot solve the equation Tju = f for f € W(T2). When & (T}) is closed, 


Hinherits the following metric structure from A: 
[fl] = inf{||f + gllalg € #(Tr)}, 


with which His complete. Furthermore, as we noted before, 


MN (T2)=# (T1) ® (W (T2) 9 #(T1)") =A (T1) © YW (Q). 


Thus His isomorphic to VY (Q) in this case. Together with Proposition 5.3.7, we obtain: 


Proposition 5.3.8 0 is essentially positive if and only if # (T1) and & (T2) are closed and H is finite dimensional. 


For 1<q<n-—1, let 


Q,(u,v) = (By, Bye) + (3,14, 3,1) 


_* 


with Dom (Q,)=Dom (dq) Dom(0,,). The bottom and top degree cases are defined similarly with 
Qo(u, v) = (Bou, a) and Q,(u, v) = (3,u, 9,0). By Proposition 5.3.3, Q, is a densely defined, non- 


negative, closed sesquilinear form and the associated self-adjoint operator is the O-Neumann Laplacian. More 
precisely, we have: 


Theorem 5.3.9 The 0-Neumann Laplacian O, : L? 
such that 


(2) > Lio 0 (2) is the densely defined self-adjoint operator 


Dom (O"/?)=Dom (04) N Dom (3,1) 


and 


Qq(u, v) a ri ae): 


Furthermore, u € Dom (Hq) ifand only ifu € Dom (Q,) and there exists ag € Lio 0 (2) such that 


Q,(u, v) = (9,0) 
for allv € Dom (Q,). 


5.4 Pseudoconvexity 


Pseudoconvexity is a central concept in several complex variables. In this section, we will review the rudiments of 
this concept. Interested readers can find a more extensive treatment in [Kr01]. One of the most striking differences 
between one complex variable and several complex variables is the Hartogs extension phenomenon, of which the 
following is the simplest example: 


Example 5.4.1 Suppose f (21; 22) is a holomorphic function on 
Q= {(z1, 22) E C?||z1| < 2, |z2| < 2}\ { (21, 22) E C?||z1| <1, |z2| a 1}. Then f has a _ holomorphic 


extension f to = { (z1,22)l|21| < 2, 29! < ah. 


Proof: For |z1| < 1, expanding f(z1, z2) as a Laurent series in z, we have 
SO < 
f(z, z2) = D> aj(z1)24, 


j=—00 


where 


. — 1 f(z¢) 
a;(z1) = a5 I... Cit d¢. 


Since f(z1,z2) is holomorphic on {|z1| <2} when |z2| = 3/2, it follows that aj(z1) is holomorphic on 
{|z2| < 2}. Moreover, since f(z1, z2) is holomorphic on {|z1| < 2} when 1 < |z2| < 2, we have a;(z1) =0 on 
{1 < |z1| < 2} when j < 0. Hence a;(z1) = 0 on {|z1| < 2}. Therefor, fis holomorphic on 2. 7 
Convexity is not preserved under a biholomorphic map and whence not a natural notion in several complex 


variables. Pseudoconvexity is the analogous notion in several complex variable that is invariant under 
biholomorphic maps. 


Recall that a domain 0 in R” is convex if the line segment joining every pair of points in ( also lies in Q. For a 
C’-smooth real-valued function p, the real Hessian of p is the symmetric matrix: 


R;,,\ — { ®o(P) 
A; (p) _ (ee ) ee 


For economy of notations, we will identify the tangent space T,(R”) with R”. 


Proposition 5.4.2. Let 2 = {p < 0} be a bounded domain in R" with C*-boundary. Then Q is convex if and only 
if 


n 0p 
HEE) = > ope Gh 20 VEC THO) 
j,k=1 
(5.22) 
where T,(bQ y= fe ER" ee i =o} is the real tangent space of b2 at p. 


Proof: We first prove the necessity. Let p € b{2. After a translation and a rotation, we may assume that p is the 
origin and the positive x,-axis is the outward normal direction for b(2 at p. Thus 


". 07p(p) 


’) 
Ax ,;0x;, ; 


xx, + o( |x 


* el 


Proving by contradiction,we assume that 


(8? p(p)/Ox;0xn)" 


is not semi-positive definite. After a rotation, we may assume that the above matrix is diagonalized and 


n—-1 n—-1 0" p(p) ‘ 8 p(p) 
p(x) = an +30 my\" +50 Dn.On tien t I Ger Tn +o(|z]?), 
j=l ga, ee 
where one of the Ajs, say A,, is negative. However, this leads to a contradiction as we will then have (—e¢,0,...,0) 
and (¢,0,...,0) in for sufficiently small ¢ > 0 but (0,..., 0) is not in Q. 
The proof of sufficiency is left as an exercise. P] 


A domain 0 is said to be strictly convex at a boundary point p if the inequality (5.22) is strict when 
é Ps 0. A domain is strictly convex if it is strictly convex at any of its boundary points. 


Exercise 5.4.3 Let 2 = {p(x) < 0} CC R” is a convex domain with C’-boundary. Let py = pe“ Iz” Show 
that 2. = {fu < €} is strictly convex when M is sufficiently large and ¢ is sufficiently small. Use this to prove 
the sufficiency in Proposition 5.4.2. 


Definition 5.4.4 Let 2 = {p < 0} Cc C” be a bounded domain with C’-boundary. Then 0 is Levi-pseudoconvex at 
pe bNif 


3” o(p) 
Oz;0Z% 


H,()(é6) = 32 bk, > 0, 


jk=1 
“. 0 

ve € T)°(60) = ‘é eC) 5, PE = o} 
jal 0%] 


It is strictly pseudoconvex at p if the above inequality is strict when = 0. The domain 0, is said to be Levi- 
pseudoconvex or strictly pseudoconvex if it is Levi-pseudoconvex or respectively strictly pseudoconvex at every 
boundary point. The Hermitian matrix 


2 p(p) 
H,(p) = ( ae ) 
is called the complex Hessian of p. 


REMARK 5.4.5 (1) The notions of Levi-pseudoconvexity and strict pseudoconvexity do not depend on the choice of 
local holomorphic coordinates. More precisely, if b§2 is Levi-pseudoconvex (respectively, strictly pseudoconvex) at 
p € 692 and w = W(z) is a biholomorphic map from a neighborhood U of p into C”, then ¥(b22N U) is Levi- 
pseudoconvex (strictly pseudoconvex) at W(p). This is a consequence of the following functorial property of 


vectors and forms: Let € = ¥%(€), # = W(p), and p = p°Y~!. Then 


H,(B)(E, €) = O9R(E, €)(B) = Op(E, €)(») = Hp(v)(E, 8). 


(2) Every bounded convex domain © with C’-smooth boundary is Levi-pseudoconvex. The converse is of 
course not true as a smoothly bounded planar domain is always Levi-pseudoconvex but not necessarily convex. 
The proof of the first statement is as follows. Let € € C”. Write €; = t; + 7s;. Then 


and the convexity of implies that the real Hessian 


HBa\s) = 5A + bo Se eae PP) 


jk=1 j,k= 


= 2Re (35 3, a tae 42 Se o P) eg, >0 


Now let € € T!(b2). Replacing € by 7€ in the above inequality and then adding the result to the above inequality, 
we then obtain 


n do 
> 
py 8z;0Z. 3z05, ot 2 9 


Proposition 5.4.6 Let 2 CC C” be strictly pseudoconvex with C? -boundary. Then 


1. There exist a neighborhood U of b22, a defining function of Q, and a positive constant c > 0 such that 


0° p(z) 
0z;0Z% 


E,€, > cl€l?, VzeU, VEE C". 


g 


j,k=1 
(5.23) 


2. For any p € b22, there exist a neighborhood U and a biholomorphic map from U into C” such that 
bQN U is strictly convex. 


Proof: (1) Let 6 = p + Mp?. It follows from direct computations that 


0*p(z) = n 8? p(z) - 


From the strict pseudoconvexity of 0, we know that there exist a neighborhood U of b2, a positive constant c > 0, 
and a sufficiently small ¢ > 0 such that 
(20 g ) 


“. 0? p(z) , = 
» Dz.Bz, ciek = © 
rs es a 
By choosing M sufficiently large, we then obtain (5.23). 

(2) Let p be a defining function of 0 that satisfies (5.23). After a translation and a unitary transformation, we 
may assume that p is the origin and the positive x,-axis is the real outward direction. Thus, 


2 <e 


€/?, Vz €U,€ € C” with él? 


=~ 8*p(p) =~ 8°p(p) 2 
p = Rez,+Re ZiZk | + —7;%+0(|z|"). 
2. 02,02 J Ps 02,02 : 
After a change of holomorphic coordinates in the form of w = ¥(z): 


" Foz) _ 
Az;0z, 7” 


jk=1 


we have 


e =~ O*p(w) 
p(w) = ae 2 Ow OW, 


WjWi+0(|w”). 


It follows from (5.23) that 


p(w) 

OwjOw, 
is positive definite. It then follows that after a unitary transformation in the w-variables, the quadratic term in the 
Taylor expansion of # is 


2 
Wil", 


ys 
jel 


where the /,s are positive. This implies that b2 is strictly convex at p. After possible shrinking of U, we then 


obtain that 692M U is strictly convex in the w-coordinates. Py 


Definition 5.4.7 Let f : 2 — [—oo, 00). It is an exhaustion function of O. if {z € Q| f(z) < c} CC Q for any 
c € R. It is plurisubharmonic on 0 if 


1. fis upper-semicontinuous on QO; 
2. For any z € Mand € € C", u(t) = f(z + t€) is subharmonic on {t € Clz + t€ € OQ}. 


We say fis strictly plurisubharmonic at p € 92 if there exist a neighborhood U of p and a positive constant c > 0 
such that f(z) —c|z|? is plurisubharmonic on U. It is strictly plurisubharmonic on Q. if it is strictly 
plurisubharmonic at every point in 2. 


Exercise 5.4.8 Let f € C?((2). Show that fis plurisubharmonic function if and only if its complex Hessian H (z) 
is semi-positive definite at any z € 2 and fis strictly plurisubharmonic if and only if H f(z) is positive definite for 
any z € 22. 


Definition 5.4.9 (1) A domain 2 Cc C” is called pseudoconvex if it can be exhausted from inside by smoothly 
bounded strictly pseudoconvex domains. More precisely, there exists a sequence of smoothly bounded strictly 
pseudoconvex domains 2; such that 2; C 2j41 and U7 2; =. (2) A domain 2c C” is a domain of 


holomorphy if there is no domain such that Q g Q and every holomorphic function on Q extends 


holomorphically to 2. 


Evidently, every domain in Cis pseudoconvex. Indeed, every bounded domain in C with C’-boundary is strictly 
pseudoconvex according to Definition 5.4.4. In this case, T}°(bQ) is trivial. 


Theorem 5.4.10 Let Q be a domain in C". Then the following statements are equivalent: 


1. QO is pseudoconvex. 
2. — log 6(z) is plurisubharmonic on Q, where 6(z) is the Euclidean distance from z to the boundary bQ2 of 


3. 0, has a smooth strictly plurisubharmonic exhaustion function. 


Furthermore, if Q has C?-smooth boundary, then the above statements are equivalent to Q being Levi- 
pseudoconvex. 


We skip the proof and refer the interested reader to [Kr01]. 


Theorem 5.4.11 Let QC C”. Then 2 is pseudoconvex if and only if Q is a domain of holomorphy. 


The proof of sufficiency is not very difficult. The opposite direction is usually referred to as the Levi problem, 
and it was solved in the 1950's by Oka. We again refer the reader to [H91, Kr01] for detail. 


ee 
5.5  Hormander's L2-Estimates 


In this section, we present Hérmander's L?-estimates for the 0-operator through the lens of spectral theory of the 0 
-Neumann operator. The following main theorem in this section is due to H6rmander ([H65]). 


Theorem 5.5.1 Let Q be a bounded pseudoconvex domain in C” with diameter D. Then 


inf o(Og) > pr» l<q<n-1l. 


(5.24) 
Consequently, for any f € NW (Oq), there exists u € Dom (04-1) such that 
du=f and lull? < 7 \/FI?. 
(5.25) 
We divide the proof into several steps, presented in the following subsections. 


5.5.1 The Morrey-Kohn-Hormander formula 


We first introduce some notations. For (0, g)-forms 


=o and v= oe 


lJ|=a lJ|=4 
on a domain 0 in C’ and a real-valued function y € C?(2), we write: 


I n ekg 


We also write: 


Ou;z 2 
OZE 


|= =I 


Let LD? 


(0,4) (2, e-*) be the space of (0, q)-forms with weighted L?-coefficients equipped with the norm 


i 
||e||2, = » i uz|\7e ° dV, 


\J|=q 


(Q,e-%) > L? 


and the inner product (u,v), = (u, ve?) Let 54 be the adjoint of 0, : Lig (o,gt1 


) (2, e-*). 


Since 


we obtain that Dom (0°)=Dom (3,) and 


* 


5, =e?" ¥ = 5" + (Ty) 4. 
(5.26) 


The following integration by parts formula, due to Morrey-Kohn-Hérmander, plays an important role in the L?- 
theory of the 0-problem. 


Theorem 5.5.2 Let 2 = {z € C” | p(z) < 0} CC C” be a bounded domain with C?-boundary and let p be a 
defining function of Q such that |Vp| =1 on 692. Let y € C?(Q) be a real-valued functions. Then for any 
we Coo, gi2 2) Dom (0°), 


r= _x* - aa) _ 
Jule + leulg =f Hplwe ras f (|e + H,(u) Je ¥ dV. 


(5.27) 


Proof: We will (5.27) step by step. We first prove it for the simplest case when y = 0 and q = 1. (This case is due 


to Morrey. The formula for a general g is due to Kohn. With the weight g, it is due to Hérmander.) In this case, 
since 


- Our a) 
du= (3 - = dz; \ dZ,, 
Poy OZ; OZ, 


we have 


= Ou Au, ||? 
IBul? = >| oe ae 

Our 

Oz; 

_ . “i du; ||? 

a 1 OZ, 


and 


+| Ou; 


Ou, Ou; Ou; Our 
OZ, OZ; OZ, O2;,’ OZ; 
Ou; Our 
OZ, OZ; 


(5.28) 


Ou; 


3 Oz; 


|°ull? = 


> os, = a 
jk Oz; : Oz, 


(5.29) 


Applying integration by parts formula (5.8), we have 
Ou; Our ) = Oru; i Uj — Op 
=, = = ,UR) + —Up 5 dS 
( OZ, ? 02; ( O0zj0Zp ‘) 60 OZk k Oz; 


= (Hg) + f (eu OR es tie =e.) as 
of ak b2 OZr Oz; Oz; OZE 


Summing up from j, k = 1 to mand using the boundary condition 


we then have 


(5.30) 


We now play the “Morrey trick” to convert the boundary term into a desirable form: Since .S uj0p/Oz; = 0 on 
j=l 
b2, we can write 


. 0, 
diaz = fe 
jal 
(5.31) 


for some f € C! near bM2. Differentiating both sides with respect to Z,, multiplying the result by wz, and then 
summing up over k = 1 to n, we have 


n n Ou; fa) a2 a - 
(oR a a) = ou (egh + £32). 


k=l \j=1 k=1 


Therefore, on 62, we have 


. — Op _ ae Ou; Op 
dS ujtte B20 ye eae Oz; ° 
k=l j,k=1 
(5.32) 
Combining (5.28), (5.29), (5.30), and (5.32), we then obtain (5.27) when y = 0 and q = 1. 
We now prove (5.27) when q = 1 but with a general real-valued yp € C?(2). We write 
0; = 0/0z;, oP = —e?0je " = —0; + Ojy. 
For the economy of notation, we also write 
05 = 950, py = O40, 
and so on. It is easy to check that 
[Ox, 37] = Pak 
(5.33) 


As in the proof of the previous case, we have 


Bul + Dell? = Vell + So ((afu),afus — (Fy Bjus) 


j,k=1 


(5.34) 


It follows from the integration by part formula (5.8) and commutative identity (5.33) that 
(Bxu;, yur ) = (uj, 0f5;ux) +f uj0ju,0,pe ° dS 
g g bQ 

= (uj, 9;0fux) _ (a;D.pu;, ux) +f ujO;UpOKpe ° dS 
g g bQ 

= (aPuj, Ofur ) _ (a;d.puj,ux) 
y g 

+f uj0jU,Onpe ? as— [ uj0UpO;pe * dS 
bQ bQ 


Summing up over j,k = 1 to n and using the boundary condition S °u;0;p = 0, we then obtain that the last 
boundary term vanishes. Using the Morrey trick (5.31) to the first boundary term, we then obtain formula (5.27) 
when q = 1. 


We now prove Theorem 5.5.2 in its full generality. We first fix some notations. Suppose A and B are tuples of 
integers between | and n of same length. We let eS be the sign of permutation from A to Bif they contain the same 
set of integers and be zero if otherwise. Throughout the proof, we will use the capital letter J and L to denote 
strictly increasing q-tuples, K strictly increasing (q— 1)-tuples, and M strictly increasing (q+ 1)-tuples. The 
lower case letters j,k, and / will denote integers running from | to n. If J = (j1,...,jq), then jJ denotes the 
(q + 1)-tuple (j, j1,-.-,Jq)-. Note that 


Let 


Then 


It follows that 


/ / — U 
|u|? = ~ ~ ~ Ou; Our, cid gl _ Ouz uz cid 
. / Oz; Oz) M~M > ; Oz; Oz, ~IL 


M \ TL Gl JL id 
/ / 

= Our our ei ey Ous ou, eid 

zx Oz, “IL Oz. Oz “IL 

JL jal C75 TL gett O75 
/ 

— Ouz _ Oulk OU;K 

a i >, > >. OZ 
J jZI Oz; K jAl Oz; 
! 2 ! 

->>d |e] -L ye Sess 

_ > >. Oo 
Ij Oz; K ji Oz; 


(5.36) 


Since 


we have 


x OuRK 9 
5) uU Duns ii 
», kl Oz, 
(5.37) 
Therefore, we have 
Full + O,ull2 = l|Veulli? 
/ 
+ SS oe (— (june, Byujx.) + (OfurK, af wx), ‘ 
K ji " 
(5.38) 


The rest of the proof follows along the same lines as in the previous case, modulus index complications. Details are 
given below. It follows from the integration by part formula (5.8) that 


(Fjunx, Biujx ) = (unc, AF Biuzx ) -{ uk OujK p;e °dS. 
yg yg b2Q 


(5.39) 
Using commutative identity (5.33), we then obtain that 
(ui, 9 Bru;x ) = (wrx, F197 u3x ) = (winx, patiK ) 
y y y 
= (aPux, OFujx) 
yg 
= / UlKO; UjK Pl e °dS — (wi, P(4ik ) : 
b2 y 
(5.40) 
Notice that since u € Dom (0°), 
S> ujKpj=—0 on bs. 
j 
(5.41) 


We now repeat the Morrey trick: From (5.41), we have 


Bike; = fp 
j 


for some C! function f in a neighborhood of b. Applying 0, to both sides, multiplying them by wx, then 
summing up over J, we then have 


) UK OUjKP; = — ) UWKUjK PG on 692. 
ji ji 


(5.42) 


Back-substituting (5.40) into (5.39), then (5.38), and using boundary conditions (5.41) and (5.42), we then obtain 
the desirable (5.27). Py 


The following formula, sometimes referred as the twisted Morrey-Kohn-Hoérmander formula, is a 
consequence of Theorem 5.5.2 (compare [BSt99] and [Ch13]). 


Corollary 5.5.3 Let Q = {z € C"|p(z) < 0} Cc C” be a bounded domain with C*-boundary and let p be a 
defining function of Q such that \dp| = 1 on bQQ. Leta, py € C?(Q2) be real-valued functions with a > 0. Then for 
anyu € Coo (2) Dom (d°), 


| va5ul2 + |vad,ul2 = [att (ue vas 


+f (alr) +aH,(u) — H,(u))e~? dV + 2Re ((8a) ou, du) ; 


y 
(5.43) 
Proof: Let ¢ > 0. Applying Theorem 5.5.2 with g replaced by , = y — log(a + €) and using the identities 
05 = 9° + (0%) 1= 9, —(a+e) ‘(Oa) 4 
and 
H5(u) = H,(u) — (a+) 'Ha(u) + (a +e) *|8a)* su)”, 
we obtain 
Wa + € OullZ + ||\VWa+e O,ull? = [@ +e)H,(uje ° dS 
as 2 — * —* 
if / ((a +e)|Vu]" + (a + ©)H,(u) - H,(u))e* dV + 2Re ((Ba) au, Bu) . 
7) y 
Letting ¢ — 0, we then obtain the desirable identity (5.43). a 


The following estimate is due to Catlin [Ca87]. The proof presented here is from [BSt99]: 


Theorem 5.5.4. Let Q = {z € C"|p(z) < 0} CC C” be a bounded domain with C*-boundary and let p be a 
defining function of Q such that |\dp| = 1 on b22. Letb € C?(Q) withb < 0. Then 


Qiu, uw) S [etHalw) dV, Yue Dom(a')n Cioq (2). 


(5.44) 


Proof: Taking a = 1 — e° and y = 0 in the twisted Morrey-Kohn-Hormander formula (5.43), we have 


JVaBul? + |vad"ul? > fal Pul*av — ane { Ba)’, 3°u 
2 


I n 
bf ab Ob. _db a 
+> fee + BB) unin av. 


K j,k=1 
(5.45) 
Applying Schwarz inequality to the second term on the right hand side, we have 
| VaDul? + [9"ul? > f alVul?av 
Q 
U n 
HOY f baru stn aV. 
K jk=1/2 
(5.46) 
The desirable inequality (5.44) then follows. a 


Proposition 5.5.5 Let Q be a bounded pseudoconvex domain with C?-boundary. Let D be the diameter of 0. 
Then 


Q,(u,u) > Sllull?, Vu € Dom(d*)n Coq (2). 


(5.47) 
Proof: Let p be a point in 0. Let b = —1 + |z = p70". We then obtain (5.47) from (5.44). 2 


In the next subsection, we will transform (5.47) from an apriori estimate to a bona fide one: namely, we will 
establish (5.47) for all wu € Dom (Q,). This is done by applying the Friederichs lemma (see Lemma 5.5.6 below). 


5.5.2 The density lemmas 


Density is a subtle issue in the 0-Neumann problem. This issue was overlooked in some early literature and was 


addressed by Hérmander in [H65]. (We refer the reader to [H03] for a fascinating history about the 6-Neumann 
problem, especially the density issue in the problem.) The presentation here is similar to that of [CS99]. 
We begin with the Friedrichs Lemma. Let x € C'°(IR™) be a radial symmetric function such that x > 0 on 
Jz] <1, x =O on |a| >1, and / xdx=1. Let x-(#) = (1/e™)x(a/e) denote the standard Friedrichs 
R™ 
mollifiers. Before we proceed to prove this important lemma, we first recall the following simple facts. 


1. The product of a Lipschitz continuous function and the first order derivative of an L?-function is a well- 
defined distribution: 


ab,(f\v) =~ f fD,(av)dz, ve BR”), 
where D; = 0/02}. 
2. Let 6 € BU). Let g,(-) = o(- — y) and 
Us = {y € U|Supp¢d, C U}. 


Let T € 9'(U) be a distribution. Then y +> T(¢,) is C~(U4). This is due to the fact that d,(x) = ¢(x — y) 
depends smoothly on y. 


3. Let T € J'(R™) and let g € C§°(R™). Then for any test function ¢ € A(R”), 


(PgN() = T(¢-*4) = f e(a)o(e) ae 


where g(x) = g(—2) and t(x) = T(g,) = T(g(a — -)). Note that t(x) € C(R™). Thus a convolution of 
a distribution and a compactly supported smooth function results in a smooth function. 


Lemma 5.5.6 (Friederichs). Let f <¢ L?(IR") be compactly supported in an open setU Cc R™. 
1. Let a be a Lipschitz function on U. Then 


aD;(f*x-) — (aDj(f))*x- +0 in L?(R™). 
(5.48) 


2. Let L = SajDj + ao where a, 1 < j <™m, are Lipschitz on U and ag € C(U). If Lf € L?(R™), 
j=l 
then 


frye f, Lf*xe) 9 Lf in L?(R™). 
(5.49) 


Proof: (1) Note that 


and 
(aDjf)*xe = (aDjf)(Xe(x — -)) 
=~ FOF (aly)xcle— way 
= - fy) dveeg eet 
= | Sle ew) (Sie -(w — ey)x(y) — a(x ey) 2 $y) ay. 
Therefore, 
aDi(Fx2) — (aD,f)*xe =~ [ F-(@—w)sle ~ wrelv) dy 


) 
+ [ (ale) — ale ~ey)) fle — ev)? Ea 
Since ais Lipschitz continuouis on U, we have that 
ja(x) —a(a—y)|< Mly| and |a/dy;| < M 
almost everywhere on the support of £ Therefore, by Young's inequality, we have 


OXe 
Y dy, 


ay) < Ollfll. 


laD,( f*xe) — (aD; f)*xell < MIlf|| ( fretvs f 


Thus we have now proved that there exists a positive constant C such that 
l|]aDj(f*Xe) — (@Djf)*xell < Cllfll, VF € L°(R™). 
(5.50) 


We now show how this implies (1). Let fy, € C°°(IR™) be a sequence of functions such that f, + f in L?(R™). 
Then 


]aDj(F*x-) — (aDjf)*xell_ < |laD (CF — fe)*xe) — (@Di(f — fr))*Xell 
+ |laDj(fn*Xe) — (Dj fe)*Xell- 


By (5.50), the first term on the right hand side tends to 0 as k — oo. The second term tends to 0 as e — 0 because 
fr € C2(R™). 

(2) By the first part, we know that ||/L(f*x-) —(Lf)*xe|| 0. Since Lf € L?(R™), we have 
|(Lf)*x- — Lf|| > 0. We then obtain (5.49). 7 


In proving the density of C*(2)MDom(d") in Dom(Q,), We will localize the problem near the 
boundary of 0 by using a partition of unity. The following formula is convenient for this purpose: 


Lemma 5.5.7 Let yj; € C*, 1 < j < m, be partition of unity such that \~ x; =1onQ. Then 
j=l 


j=l j=l 


Proof: By straightforward computations, we have 


(I(x;u), B(x;u) ) (x38u, Bu) + 2Re (x59; Au, du) + (8x; Au, Ox; w) 


yp 


and 


+ ((5x;) au, (Bx;) su) —2Re 


yp 


where (Oy) denotes the (0, 1)-vector that is dual to Oy with respect to the ambient metric A and 4 denotes the 
contraction operator. Summing up and using the fact that SS Xj Ox j = 0, we then have 


* * 


YF = Vm &O 6 (x ju, xju) & = Qp(u,u) + d(x Aas Ones u), a (Gx) tH, (8x3) ),) sie 
j=l 


We can now state and prove the following density theorem due to Hérmander ([H65]). = 


Theorem 5.5.8 Let k be a positive integer. Let 2 = {p < 0} be a bounded domain in C” with C**!-boundary. 
Then Dom (3,1) N C*(Q) is dense in Dom (Q,) in the graph norm ||u||q = (\\u\|? + Q,(u,u))?. 


We divide the proof into several steps by first proving the following lemmas. 


Lemma 5.5.9 (1) For any domain 2. CCC", Dog-1)(2) is dense in Dom (3,1) in the graph norm 
a* 1/2 a* . Pet irs A 

flee = (IFIP? + lO" F117)". (2) For f € Lio g-1 (2): f <€Dom(0,,) if and only if f € L’, )(C") where 

f =f onQand f =0 onC\Q. 


2 
(0,4) 


Proof: The first part follows directly from the fact that 0 . being the adjoint of maximally defined operator 0, is 


minimally defined (see the Remark after Definition 5.2.4). Thus Yo,q-1)(2) is dense in Dom (0,,,) in the graph 


norm ||f||5+ = (||? + Wor sf2) 7. To prove the second part, we let f € Dom (5.1). Set f = f onQ and f =0 
on C"\Q. For any y € Dog-1)(C”), 


7 5p) ., =, 5p) , = (9f, 9) g- 


(2) with 0f € L? (C”). Then for any 


It follows that ¥f € Lio (C”). Conversely, suppose f € Li (0,¢-1) 


Q) 0,q-1) 
pe Bog-1) (2), 


«7 dp), ay, 5p), = (9, ®) os = (8f, ~) a: 


Hence f € Dom (0,1). = 
Lemma 5.5.10 Let 2 CC C” be a bounded domain with C!-boundary. Then CE gl?) is dense in Dom (Q,) in 
the || - || g-norm. 
Proof: Let f ¢ Dom (Q,). Let {25} 5 be a covering of 2 with the following properties: 

1.29 CC 2. 

2.2;Nb2 4 0,1<j<m. 


3. There exist a sufficiently small constant ¢) and (outward) unit vectors ny € C” such that for any ¢ with 


0O<E< €, 
m aA - 
UR, {2 — 2enjlz € 2 2b UM Cc 2; 
and 
m mijze Qa, nabun at 
Um 12+ 2en;|z € 25ND UM DD 2: 
where 


> = {z € A\dist (z,b2)>e} and NF = {ze C"ldist (z, 2) < e}. 


€ — 
Let {,} be a partition of unity subordinated this covering. We may assume that Supp; CC 92;. Let 


m 


fe(z) = Wo(2) F(z) + S> vy(2) f(z — 2en)). 


j=l 
(5.51) 


Then Of, 9f. € L?(Q;). Furthermore, Of, > Of and 0f. > Uf on L?(Q). Regularizing f, with the Friederichs 


mollifiers (here the convolutions of forms with the Friederichs’ mollifier v, are done component-wise), we have the 


desirable f. = f.*x5, € Coy (2) with 6, > 0 sufficiently small, depending on e, such that f. — f in the || - ||g- 
norm as ¢ — 0 by Lemma 5.5.6. a 


Let 2 CC C” be a bounded domain with C'-boundary. Let p € C! be a normalized defining function of 0. 
For f € Li, (2), write 


f’ =OpA fy, iT hoe ae 


where 


ie ! no 9 
fv := (8p) af= > (x: te) dix. 
\K|=q-1 73 
Then f’, f7 € Lio (2): Note that for f € Cog (2): 


(3p) af? =0. 


—* 


Thus f? € Dom (0,.1). 


Lemma 5.5.11 Let 2 C C” be a bounded domain with C'!-boundary. Let f € Dom (Q,). Let f” be extension of 


f’ to 0 outside of Q. Then (1) Of” € Log1y(C”); (2) f7 € Dom (0,1). 


Proof: We first prove (1). Notice that since p € C!!, we have f” € Dom(0). From Lemma 5.5.10, we may 


assume f € Cp) ({2). By Lemma 5.2.7, for any y € Doo,g+1)(C”), we have 


It follows that ary = Of” € L*(C"), where Of” is the extension of Of” to 0 outside of 0. The proof of (2) is 
similar. 7 


We are now in position to prove Theorem 5.5.8. Let 0 be a bounded domain in €” with C’ K+] boundary. Let p 
be a normalized defining function of ©. Let {Qj} 4 be a covering of and let {Pi} iro be a partition of unity 
subordinated to the covering as in the proof of Lemma 5.5.10. Let f € Dom (Q,). For sufficiently small ¢, set 


m 


FY (2) = W(2) F(z) + Sov (2) F(z + Ben). 


f=1 
Note that Suppf” Cc ;. Let 
tex = fo Xs. € Bog (2), 


(C”) and by Lemma 5.5.11, Of” € L? (C”). It 


when 6, is sufficiently small. By Lemma 5.5.9, of” € Lio (0,q+1) 


follows from Lemma 5.5.6 that 


.q-1) 


fev — f” inthe ||-||g -norm. 
(5.52) 


Let f;(z) = f(z- 2en,) and let 


m 


FP Q=PAOL D+ df)”. 


j=l 


Then 0 f7*(z) € L? 


(ogra (22) and 3fz* € Lt (QF). let 


(0,q-1) 
fer(z) = Ar ae + LHe (ii Xa. ( (z))’, 


where 6, is sufficiently small. Then f,,-(z) € Clo, gi2 2) 7 Dom (0°). Furthermore, by Lemma 5.5.6, 


fer —> f" inthe ||-||q -norm. 


(5.53) 


Therefore, fe = fe» + fer € Ch foi) (2 2) M1 Dom (0*) and converges fin the || - ||g-norm as € — 0. This concludes 
the proof of Theorem 5.5.8. 

We now prove Theorem 5.5.1. Let 2; be a sequence of smooth bounded pseudoconvex domains such that 
2; CC 541 and UP 2; = . By Proposition 5.5.5 and Theorem 5.5.8, we have 


ODS f € Dom(Q%), 


where Q: is the quadratic form associated to the @-Neumann Laplacian on 2;. Let f € Li o(®) and Of =0. 


By Proposition 5.3.6 (and its proof), there exists u; € Lio, = (Q; ;) such that 


poet 2 

Ou; = fon 2;, and esl, < ve 

Passing to a subsequence, we may assume that u; converges to some u € Lio, ge (2) weakly. It follows that 
du = fon Mand |lul? < 22 |) f\l?. 


Since the above estimates hold for any 1 < g < n — 1, we obtain Theorem 5.5.1 by applying Proposition 5.3.6 in 
the reverse direction. 


5.6 Hearing Pseudoconvexity 


In this section, we establish the following characterization of pseudoconvexity via spectrum of the 0-Neumann 


Laplacian. Thus in Kac's language, one can “hear” pseudoconvexity via the O-Neumann Laplacian. The 
presentation follows closely those in [Fu05, Fu08]. 


Theorem 5.6.1 Let Q be a bounded domain in C” such that int (cl (92)) = 2. Then the following statements are 
equivalent: 


1. QO is pseudoconvex. 
2. inf o(Og) > 0, foralll <<q<n-1. 
3.info,(O,) > 0, forall <q<n-1. 


We first introduce some terminologies. The Dolbeault and L?-cohomology groups on © are defined respectively 
by 


{LEC (M)Iaf=0} 
{8,.9l9eC%,_(@)} 


H4(Q) = 


and 


{ FeLi, (|Faf=0} 


Hot Q)= . 
1? ( ) {8,-1919eLig 1 (2)} 


These cohomology groups are in general not isomorphic. For example, when a complex variety is deleted from 0, 
the L?-cohomology group remains the same but the Dolbeault cohomology group could change from trivial to 
infinite dimensional. As noted in the paragraph preceding Proposition 5.3.8, when # (04.1) is closed in Ling (2), 
Hy H(Q) = W (Oy). 

The implication (1) = (2) is Hérmander's L?-estimates of the O-operator [H65], and it holds without the 
assumption int (cl (2)) = 92 (see Theorem 5.5.1 in the previous section). The implication (2) = (8) is trivial. 

It remains to prove the implication (3) = (1). This is a consequences of the sheaf cohomology theory dated 
back to Oka and Cartan (cf. [Se53, L66, Siu67, Br83, 082]). An elementary proof of (3) implying (1), following 
[Fu05], is given below. The proof uses ideas from sheaf cohomology theory in [L66]. When adapting Laufer's 
method to the L’-settings, one encounters a difficulty: While the restriction to the complex hyperplane of the 
smooth function resulting from the sheaf cohomology arguements for the Dolbeault cohomology groups is well- 
defined, the restriction of the corresponding L? function is not. This difficulty was overcome in [Fu05] by 
appropriately modifying the construction of auxiliary (0, g)-forms. In showing that (3) implies (1), we will actually 
prove the following statement: 


Proposition 5.6.2 Let Q be a bounded domain in C” such that int (cl(92)) = 2. If dim H (0) < oo for all 
1<q<n-1, then Qis pseudoconvex. 


To illustrate the idea of the proof, we first show the case when n = 2. Proving by contradiction, we assume that 
Q is not pseudoconvex. Then there exists a domain ND such that every holomorphic function on 0 extends to 2. 
Since int (cl (Q)) = @, @\cl(Q) is non-empty. After a translation and a unitary transformation, we may assume 


that the origin is in A\cl (2) and there is a point 2 in the intersection of z,-plane with © that is in the same 


connected component of the intersection of the z)-plane with Aas the origin. For any a EN, let 


an (—21dZ2 + ZdZ}). 


Ua = |z|2(2+2) 


Since the origin is not in 2, u, € C (01) (2). In particular, wa € Lia (2). Furthermore, one checks easily that 


OUq = 0. In fact, we have 


Let Mbe an integer such that M > dim H (0,1) (2). Then there exists a nontrivial linear combination of the u,'s 


M 
Uu= S CaWa 


a=1 


such that [u] = [0]. Namely, there exists an v € L?(2) such that Ov = u. Let 


sat] 


M 
F(21, 22) = 210(21, 22) + > Ca Sian 
a=1 


(5.54) 


Then OF = 0. Notice that since 0 is bounded, F € L?(). Thus Fis holomorphic on . However, 


= Coa! 
FO, 22) = >; atl" 


a=1 *2 


(5.55) 


This contradicts the fact the Fextends to a holomorphic function on 2 and the origin and a point z? € 2 are in the 
same connected component of Qn {21 = of. 


The above proof follows the lines of arguments of Laufer [L66] for the Dolbeault cohomology groups. In this 
case, this proof works for all n > 2 after an inductive argument, even without the assumption int (cl(2)) = 2 
because here u € C(q (2) and v € C™(), and the restriction of v to 2M {z; = 0} post no problem. While the 
above proof still works for the L?-cohomology groups, there is a subtlety needed to be addressed: The restriction of 
an L?-function to a hyperplane is not well-defined. In fact, when we obtain (5.55), we have actually utilized a 
version of the following lemma: 


Lemma 5.6.3 Let v1,...,Un—1 € L?() and let m € N. Assume that G is a continuous function on Q such that 


G(z) = y 270; (2). 


(5.56) 
Ifm >n-—1, thenG(0,...,0,2,) = 0 forall (0,...,0, zn) € 2. 
Proof: Let (0,...,0, 2°) € 2. Write 2’ = (z1,..., Z,-1). Then for a sufficiently small positive numbers a, and a), 
we have 
D(a1, a2) = { |z"| < ay} x {|zn _ 2) < az} Cc. 
For any 6 € (0, 1), we have 
1/2 not 1/2 
(/ G(62', zn) av) < are (/ v;(52', Zn) *av) 
D(a1,a2) j=l D(a,a2) 
n—-1 1/2 
<aror (/ v;(z',2n)|" wv) 
j=1 D(a16,a2) 
n—-1 1/2 
<a (/ v;(Z) 2 D(a18,a2) (2) wv) . 
jal WW 
Since m > n — 1, letting 6 — 0, we obtain from the Lebesgue dominated convergence theorem that 
/ G(0', zn)|2dV =0. 
D(a1,a2) 
Thus G(0', zn) = 0 for |zn — 22| < ao. : 


We now return to the proof of Proposition 5.6.2. The above lemma illustrates the difficulty when one tries to 
adopt the arguments of the Dolbeault cohomology groups for the L*-cohomology groups to higher dimension 


n > 2. In order for the proof to work, we need to obtain a version of (5.54) but with the term z, in front of v 


replaced by z7’ with m > n — 1. This was done in [Fu05]. We return to the case for n = 2 to illustrate the idea. 
For a,m EN, let 


(a+1)!2"* (ZZ) + 
Ua m 


mm (lea [+ ]z9|2)2? 


(—Z1dZ_ + ZdZ1). 


Note that wai = Ug. It is straightforward to check that 


5 algina+t) pa 
TD mat «J — MZ, U 3 
((esP>-+lea)")° ae 


Proceeding as before, we then obtain a non-trivial linear combination of Ugm (with m fixed and a runs from | to a 
sufficiently large integer M) and a function v € L?(Q) such Ov = u. Thus 


alz™ (a+1) 
F(21, 22) = mzf'v(z1, 22 J+ dre Tn Peepeyet 


(21 )?"+|z2/?”) 


is an L?-holomorphic function on © in a desirable form. By choosing m > n — 1, we can then make this proof 
works for all n > 2. 


We now provide a proof of Proposition 5.6.2 for all n but assuming the L?-cohomology groups H baee) are 
trivial. We again prove by contradiction and assume that 0 is not pseudoconvex. Then there exists a domain 222 
such that every holomorphic function on © extends to 2. After a translation and a unitary transformation, we may 
assume that the origin is in cl (2) and there is a point z in the intersection of z,-plane with that is in the 
same connected component of 20 {21 = 8S 24 = o} as the origin. 


Let m be a positive integer (to be specified later). Let kg = n. For any {k1,...,kg-1} C {1,2,...,n — 1}, we 
define 


u(ki, sey kg) = (q-1)!(2e, °° ‘Zkq) 


rh 


q . — 
x —1)?2,,dzp, A... A dap, A+». A dzh,; 
j=l 

(5.57) 
where rm = |z1|°" +... + |2Zn|?". As usual, dz, indicates the deletion of dz,, from the wedge product. 
Evidently, u(k1,...,kq) € Liog-1)(2) is a smooth form on C"\{0}. Moreover, u(ki,..., kg) is skew-symmetric 

with respect to the indices (k1,...,kg-1). In particular, u(ki,...,kq) = 0 when two k;s are identical. 
We now fix some notional conventions. Let K = (k1,...,k ) and Ja collection of indices from {k1,..., kg}. 


Write dZx = dz, \...\ d%x,, 2% T_ (Zh, a", and dz, = dz, A... \ dx, \ ... \dZx,. Denote by 
(k1,...,kq\J) the tuple of remaining indices after deleting those in Jfrom (k1,..., k,). For example, 


(4,6, 3, 1\(4, 1, 6\4, 6)) = (4,6, 3). 


It follows from a straightforward calculation that 


= qimz 1 _ Sd us qimzz1 n 
Ou(Ki;-+.5hy) & = ee. (rmdZK + i Zp zy dz) A (So (- ) Za, dz, ) )&= eg > f=1 
m f=] j=1 Tm LA, 


It follows that u(1,...,) is a O-closed (0,n — 1)-form. 
By Proposition 5.3.6, we have #(0q1)=/ (0q) for all 1 <q<mn-—1. We now solve the 0-equations 
inductively, using u(1,...,n) as initial data. Let v € L? 


(0n—2) (2) be a solution to Ov = u(1,...,m). For any 
k; € {1,...,n— 1}, define 


w(k1) = —mzfut+ (Aya, «++, 7\h). 


Then it follows from (5.58) that 0w(k;) = 0. Let v(k1) € Lions) (2) be a solution of Ov(k,) = w(k1). 


Suppose for any (q—1)-tuple K’ = (ki,...,kg-1) of integers from {1,...,n—1}, q>2, we have 


constructed v(K') € Lio vee (2) such that it is skew-symmetric with respect to the indices and satisfies 
ees get q a 
Bul K") =m >> (-1)?zho(K'\ky) + (-D)P Flu, ...,2\K) 
j=l 


(5.59) 


where |K'| = ki +... + kq_1 as usual. We now construct a (0,n — q — 2)-forms v(K) satisfying (5.59) for any 
q-tuple K = (ky,...,kq) of integers from {1,..., — 1} (with K replaced by K). Let 


w(K) = mS> (—1yetolK\kj) + (-1)lu(,...,n\F). 


(—1)/2@3v(K\k,) + (1) "lGu(1,...,n\K) 


(—1)’2@(m $0 (-1)'zfto(K\ ky, ki) +m > (-1)*zo(K\ ky, fi) 


=1 1<i<j j<ixq 


— (-1) TT a(1, 2. m\(K\ky))) + (I) u(1,...,n\K) 


= (1) 8m > (-1)) “zfru(1,...,n\(K\k;)) + Su(1,...,n\K)) 


q 
= (-1)8l(—m S* 2@u(ky, (1,-..,n\K)) + Ou(1,...,n\K)) =0 
jel 
Therefore, by the hypothesis, there exists a u(K) € Lio n—q-2) (2) such that Ov(K) = w(K). Since w(K) is 
skew-symmetric with respect to indices K, we may likewise choose v(KC) so that it is skew-symmetric with respect 


to its indexes. This then concludes the inductive step. 
Now let 


n—-1 


F=vu/(l1,...,n—1) =m df v((L.-. sn 1)\j) — (-1)"* 


n(n—1 
2 


) 


u(n), 


where u(n) = —2""/rm, as given by (5.57). Then F(z) € L?() and OF (z) = 0. By the hypothesis, F(z) has a 


holomorphic extension to 2. By Lemma 5.6.3, we have 


n(n—1) n(n—l) 


F(0, zn) = —(-1)"*"® u(n)(0, an) = (I) ze 


for z, near 2°. (Recall that z° € 2 is in the same connected component of 12 = of MQ as the origin.) This 


contradicts the analyticity of Fnear the origin. 
Finally, we prove Proposition 5.6.2 in its full generality. The proof essentially combine those of the previous 


two cases. For any integers a > 0,m > 1, and gq > 1, and for any {k1,..., kg-1} C {1,2,...,—1}, let 
tam (ky ++ kg) = EE =)” So 1)/2,d%, 
j=l 
Then as in the previous case, we have 
Duam(Ki,--->kq) = — etn (rm dZx 
boat taptaen ao 0's) 
f=1 = 


= mS By teva L Rigeg Mg). 


In particular, Uam(1,-..,7) is 0-closed. Our next goal is to solve the 0-equation in L?-spaces inductively with 
the (0, — 1)-forms Uaym(1,...,7) as the initial data, and eventually produce an L?-holomorphic function on b2. 
This holomorphic function has a holomorphic extension to 0. By the method of the construction, the extension has 
a singularity at the origin, which leads to a contradiction with the hypothesis. We now provide the details. 

Let S; = WV (Q,). By Proposition 5.3.7, # (Og-1)=N (Og) NSg, 1 <q <n-—1. Let M be a integer such 
that M>dimS, for all 1<q<n-—1. Fix an integer m>n-—1. Let Fy be the linear span of 
{tivia (1, Ly n);a=1,... Pe acme For any u € Apo and for any {k),...,kg_1} C {1,...,n — 1}, we set 


u(ki,...,k q-1,% -> CjUa;,m (ki,. sep hgts i) 


k 
if u= CiUa,;m(1,...,n). We decompose ¥o into a direct sum of MM”? subspaces, each of which is M- 
jUa;, ’ ) Pp P' 


dimensional. Since dim(S_1) < M and wam(1,...,n) € WY (On-1), there exists a non-zero form u in each of 
the subspaces such that 0v,,(0) = u for some v, (0) « € 12, os 9) (2). Let F; be the M”*-dimensional linear span 


of all such u's. We extend u +> v,(Q) linearly to all u € F;. 

For 0 < q <n —1, we use induction on q to construct an M"~4-?-dimensional subspace F441 of F, with the 
properties that for any we yi, there exists vy(ki,...,kq) € Lion q2)(f2) for alll 
{k1,...,kg} C {1,...,— 1} such that v,(k1,...,q) depends linearly on u; v,(k1,...,kq) is skew-symmetric 
with respect to indices K = (k1,...,kq); and 


q 
=m) (-1)izfvu(K\ky) + (-) (1, ....2\K), 
j=l 
where |K| = ky +... +h. 
We now show how to construct 41 and v,(ki,...,k,) for u € Fy41 and {k,,..., kg} C {1,...,n—1} 
once ¥, has been constructed. For any u € ¥, and any {ky,..., kg} C {1,...,n —1}, write K = (ky,...,k,), 
and let 


Wy(K) = ay (—1)’zfvu(K\k;) + (—1)7**lu(d, sen utt it). 


Then as in the previous case, 


dw,(K) = (1m > ziu(ky, (1,...;”\K)) + du(1,...,n\K)) =0, 


We again decompose F, into a direct sum of M n~@~ linear subspaces, each of which is M-dimensional. Since 
dim(S;,-¢-2) <M and Ow, (K) =0, there exists a non-zero form u in each of these subspaces such that 


Ov,(K) = wy(K) for some v,(K) € Lio ae 
we may choose v,,(K) to be skew-symmetric with respect to K as well. The subspace F,,; of F, is then the 
linear span of all such u's. 


Note that dim(.¥,,_1) = 1. Let ube any non-zero form in F,,_, and let 


(2). Since w,(K) is skew-symmetric with respect to indices K, 


F(z) =w,(1,...,n—1) = my 2M, (1,-0.4 —1\7) —(-1)"Fu(n). 


Then F' € L?(2) and OF = 0. Therefore, F is holomorphic on Q and hence has a holomorphic extension to 2. 
Restricting to z,-plane as in the previous case, we then arrive at a contradiction to the analyticity of F near the 


origin. We therefore conclude the proof of Proposition 5.6.2. a 


Positivity of the 0-Neumann Laplacian can also be used to characterize pseudoconvexity on domains with 
holes in C”. We refer the reader to [FLS] for the following and other related results. 


Theorem 5.6.4. Let 2 be a bounded domain with connected complement in C” and let D CC Q bea relatively 
compact open subset with connected complement in Q. Let Q= OND. Suppose Q has Lipschitz boundary and D 


has C*-boundary. Then both Q and D are pseudoconvex if and only if the 0-Neumann Laplacian q on (0, q)- 
forms on Q is positive for 1 <q <n — 2 and gap positive forg =n — 1. 


REMARK 5.6.5 (1) It can be proved, using the close graph theorem and the fact that 0 is a closed operator, that if 
HH} #(Q) is finite dimensional then # (041) is closed. Thus the assumption in Proposition 5.6.2 is equivalent to 
statement (3) in Theorem 5.6.1. 

(2) It follows from Theorem 3.1 in [HO4] that for a domain Q in @ with C-boundary, if inf o,( a) > 0 for 
some q between | and n—1, then Levi-form of b2 cannot have exactly n — q— 1 positive and q negative 


eigenvalues. Characterization of a smooth bounded domain in C” with inf o(L1,) > 0 for a fixed q is still unsolved. 


(3) For any domain in C”, if the Dolbeault cohomology group on (0, q)-form for a fix q is finite dimensional, 
then it is trivial (cf. [L66]). The L?-analogue of this result-even on bounded domains—remains open. The following 


related conjecture is also open: If Q is an unbounded pseudoconvex domain © in C”, then the Bergman space 
A?(Q) —the space of square integrable holomorphic functions on Q-is either trivial or infinite dimensional. This 


conjecture holds for domains in C but fails in C if the pseudoconvexity assumption is removed ([W84]; see [PZ17] 
and references therein for related results). 


Acknowledgments: This work was supported in part by the National Science Foundation grant DMS-2055538. 
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6.1 Introduction 


The connection between symmetry and artistic design has a long history (see e.g., 
Weyl] [30]). In this paper, we describe the application of domain coloring to the 
creation of symmetric designs and animations. First, we examine domain coloring 
in its original setting as a method for sketching graphs of complex-valued 
functions. Then we will describe the creation of designs in the Euclidean complex 
plane and the non-Euclidean hyperbolic upper half-plane. 

At the very end of the twentieth century, Farris [11] played a principal role in 
developing the use of domain coloring to sketch graphs of functions, f : C > C. 
The idea is to use a color wheel, a well-known tool in the visual arts. A basic 
example of a color wheel is shown at the top left of Figure 6.1. This color wheel 
is used to mark locations in the complex plane. For each value of w, the value of 
@(w) is a unique color (at least in principle). For example, on the top left of 
Figure 6.1, the values of w that are near i are colored greenish-yellow, while 
values near —1 have a light blue tint. As values approach zero they turn black, 
and beyond a certain radius they are all colored white. By composing a function 
f(z) with @, we get a function @(f(z)) that gives a color portrait of f(z). For 
instance, on the top right of Figure 6.1, a color map for w = z? is shown. Notice, 
how the colors near w = 0 cycle twice through the rainbow as we move once 
around w= 0. The connection to winding numbers is thereby made visually 
evident. Also, contour lines meet at right angles away from w = 0, just as they do 
for the function w = z. This illustrates conformality of w = z* away from the 
origin. There is a software now that produces color plots with great ease [23, 27]. 
At the bottom of Figure 6.1, we show a color plot of the function 
w = 3(z+1)(z—i)*(z-1+%)*. This plot was produced with the free 
SageMath system [23, 24]. Just these two commands were needed: 


FIGURE 6.1 


Top left: Color plot for w = z over [—2, 2] x [—2, 2]; a basic color map @. Top right: Color plot for 

w = 2. Bottom: SageMath color plot for w = 3(z + 1)(z — 7)?(z— 1+ i)*. (First published in “Creating 
Symmetry: The Artful Mathematics of Wallpaper Patterns” by James S. Walker, Notices of the Amer. 
Math. Soc., Vol 62, No 11 (Dec. 2015), pp. 1350-1354, published by the American Mathematical Society. 
©2015 American Mathematical Society). 


3* (z+1)*(z-i) *2*(z-1+i)73 
complex_plot(f, (-2, 2), (-2, 2)) 


The plot that SageMath produced clearly marks the location of the zeros at 
—1, i, and 1 —7 and their multiplicities of 1, 2, and 3, respectively. All this 
information is encoded in the number of times the colors of the rainbow are 
cycled through in the neighborhood of each zero. There are several nice examples 
of color plots at the web site created by Crone [7], including plots of branching in 
Riemann surfaces. The reader may also wish to explore color plotting using the 
free software, SymmetryWorks [27]. 


6.1.1 Pseudocode for Creating Color Plots 


We have found the most challenging part of producing color plots is the creation 
of a domain colored image using a specific color map. In Figure 6.2, we provide 
pseudocode for this procedure. It is usually a straightforward task to supplement 
this procedure with additional code for creating symmetric designs using various 
functions. For color maps, we will adopt the clever idea of Farris of using color 
photos of natural scenes, rather than a simple color wheel. The color images we 


used for the designs in this paper are shown in Figure 6.3. For those readers not 
wishing to reinvent the wheel, the software [27] can be used for artistic designs 
like the ones we discuss in the remainder of this paper. 


Domain Coloring Procedure 


CMp = loadimage(filename) %load the RGB image color map 
R = number of rows of CMp and C = number of columns of CMp 
ScaleFactor = 10 %a scaling factor for units in color map 
mCent = R / 2 and nCent = C / 2 %coordinates of image center 
xine = (2*ScaleFactor) / R and yinc = (2*ScaleFactor) / C %x, y increments 
Create color-mapped display. 
M = 1000 {increase M for higher resolution 
L=2 %increase/decrease L to zoom out/in 
incr = 2*L/M and ([x, y] = rectangular grid of points from -L to L spaced by incr 
Initialize wimg %Matrix wImg, all zeros for RGB values, M rows, M columns. 
z=x+iy Y%create array of complex values 
w = f(z) %compute values of the function f over grid 
s = real(w) and t = imag(w) %real and imaginary parts of function values 
for m= 1 to M 
forn=1toM 
%Calculate coordinates p and q for color map 
p = mCent + round(s(m,n)/xinc) and q = nCent + round(t(m,n)/yinc) 
if p>O and p<R then 
if q>O and q<C then 
wimg(m,n) = CMp(p,q) assign RGB values from color map 
end if 
end if 
end for 
end for 
Display(wImg) “Display RGB image wimg 


FIGURE 6.2 


Pseudocode for applying a color map to function w = f(z). 


FIGURE 6.3 


Images for color maps, from public domain stock images found on the Internet. Left to right: Waratah 
flower with yellow border, Reptile, Buoy. 


6.2 Designs in the Euclidean Complex Plane 


In the geometry of the Euclidean plane, there are three basic symmetry operations 
(congruences). They are (1) translation, (2) rotation, and (3) reflection. For a 
complex variable, z = x + ty, these symmetry operations are exemplified by the 
following mappings: 


1. Translation: T,: z— z+ c, where c = a + 1b is a complex constant. 
2. Reflection through the x-axis: R, : 2 Z,1e,z=x2+i1y—> x — ty. 


3. Rotation by @ around the origin: pg : z — e” z, where e? = cos + isin 8 


Any Euclidean symmetry (congruence) can be expressed as a finite composition 
of these basic mappings. This set of all symmetries of the plane is a mathematical 
group, the Euclidean group, denoted by E,. 


For creating symmetric designs, we will apply domain coloring to a function 
w = f(z) that has symmetry with respect to a group of transformations. If S is a 


transformation of the complex plane C, then the function f is symmetric with 
respect to Sif 


f(S(z)) = f(z) forallzeC 
(6.1) 


The set of symmetries of a function fis a mathematical group, S$; Our method for 


creating designs is to apply domain coloring to functions that have some pre- 
assigned symmetry group. For that purpose, the Euclidean group is too large. The 
only functions symmetric under all mappings in E, are constant functions. To 


obtain non-trivial symmetric designs we restrict to discrete subgroups of E,. We 


will examine symmetric designs created from symmetries with respect to these 
three discrete subgroups: 


1. Symmetry with respect to a lattice of two-dimensional translations 


Tmuiny: 27 Z+mut+nv allm,neZ, 


where u and v are non-collinear complex numbers. Enforcing this 


symmetry creates a tessellation over the lattice in C generated by u and v. 


2. Symmetry with respect to rotation about the origin by an nth root of unity: 


Pon/n > 2 et/N> for fixedn € Z,n# 0 


Enforcing this symmetry creates a rosette, having n-fold rotational 
symmetry about the origin. 


3. Symmetry with respect to both a lattice of two-dimensional translations and 
rotation by an nth root of unity about the origin. Symmetric designs of this 
type are only possible for rotations p27/n, with n = 2, 3, 4, or 6. We shall 
refer to a design with this symmetry as a rotationally symmetric wallpaper 
pattern. 


All of these basic examples can be modified to add more symmetry to a design, 
including various reflective symmetries. We will discuss these modifications as 
we examine these three basic symmetric designs. 


6.2.1 Translational Symmetry 


On the left of Figure 6.4, we show a design having symmetry with respect to a 
lattice of two-dimensional translations. The lattice is rectangular, and is generated 
by the non-collinear complex numbers u = 2100 and v = 17007. The positive 
integers, 2100 and 1700, are the width and height in pixels of the image used for 
the color map. That color map image is the Waratah flower image shown in 
Figure 6.3. The function f(z) that we used to create the design is, for z = x + ty, 


f(a + iy) = [x mod 2100] + i/y mod 1700] 
(6.2) 


This formula assumes that the lower left corner of the Waratah flower image is 


located at the origin in C, and uses mod-equivalence from elementary number 
theory to ensure that fis symmetric with respect to translations over the lattice 
generated by u and v. The image on the left of Figure 6.4 resembles a sheet of 
postage stamps, rather than a complex artistic design. Although if one began with 
an image of a person (say Marilyn Monroe), or a commodity (say a Campbell's 


soup can) then one could easily produce designs reminiscent of some of Andy 
Warhol's famous pictures. 


FIGURE 6.4 


Left: Design having translational symmetry over a square lattice. Right: Rosette design having 6-fold 
rotational symmetry. 


6.2.2 Rotational Symmetry 


6.2.2.1 Rosettes 


On the right of Figure 6.4, we show an example of a design having symmetry 
with respect to rotation about the origin by a 6th root of unity. In other words, it 
has 6-fold rotational symmetry. The color map used to create the image is the 
Waratah flower image shown in Figure 6.3, but without the yellow border. The 
function we used to create the design is 


fiz)=(4)4+if42 


(6.3) 


Since (e"/6) ° — 1, it follows that f(z) is symmetric with respect to the rotation 
P2r/6, 1.€., it has 6-fold rotational symmetry. We choose array values for 
z=az+zy so that the rosette design is a square image of dimensions 
1000 x 1000 pixels. The black colored regions in the design correspond to values 


of flying outside the region in C corresponding to the color map. Later, we will 
show an interesting alternative way to color these values. 


On the left of Figure 6.5, we show an example of a design having 5-fold 
rotational symmetry. It also has symmetry with respect to reflection through the 
x-axis. The color map used to create the image is the Reptile image shown in 
Figure 6.3. The function we used to create the design is 


f(z) = (2+ 34) (2° +. 2°) + i(26 Z1 + 262!) + SE (242 84+ 2°24) 


(6.4) 


FIGURE 6.5 


Left: Rosette having 5-fold rotational symmetry and reflection symmetries. Right: Rectangular view of 
rosette design with 6-fold rotational symmetry plus curved lattice symmetry. 


The terms of f(z) are grouped so that f(z) = f(z) clearly holds. Therefore, fis 
symmetric with respect to the x-axis reflection R,. To verify that fis symmetric 
with respect to the rotation p2,/5, we note that fis an example of a finite sum of 
the form: 


f(z) =k tinge’ 2" 


m=n mod 5 
(6.5) 


which is symmetric with respect to 5-fold rotation about the origin. The relation 
between Equations (6.4) and (6.5) is that we can insure symmetry with respect to 
reflection through the x-axis by requiring Q@mn = @n,m. This condition on the 
coefficients, {G@mn}, follows by considering f(z) = f(Z) in terms of the 
coefficients of the basis functions z”’Z” in Equation (6.5). 


On the right of Figure 6.5, we show another design which combines rotational 
symmetry with the mod operation used in Section 6.2.1. The function we used is 
the following composition: 


z 


Zz (1+i+4242% =u+iv = [w mod 1000] + z/v mod 1000] 


(6.6) 


We are applying the function g to the square image of the rosette design on the 
right of Figure 6.4. This is, in effect, using an infinite version of the tessellation 
design in Figure 6.4 as color map. (This is the alternative way, mentioned above, 
for coloring pixels lying outside of a finite color map.) The composite function 
satisfies g( f(e’"/°z)) = g(f(z)). Hence, the design on the right of Figure 6.5 has 
6-fold rotational symmetry. This design also has several interesting features from 
both artistic and mathematical perspectives: 


(1) 


(2) 


(3) 


For large magnitude z, we have f(z)~iz®/4. Consequently, when 
composed with g, the lattice of images of the Waratah flower are 
(approximately) pre-images of iz®/4 acting on the cells of a rectangular 
lattice of Waratah flower images (as shown on the left of Figure 6.4 but 
without the yellow grid lines). That explains the rotated, shrunken 
appearance of the flowers along a curved lattice. These flowers are most 
easily visible at the four corners of the square image. The curved grid of 
the lattice is shown clearly on the right of Figure 6.6, where we included 
the yellow borders in the Waratah flower image for our color map. 


The square framing of the design, along with the appearance of multiple 
small flowers approaching the four corners of the frame, draws the 
viewer's attention to the 4-fold rotational symmetry of the design's frame. 
This may lead to an ambiguity in the viewer's mind as to the validity of 6- 
fold rotational symmetry for the design itself. From an artistic perspective, 
we like this ambiguity. In fact, the 6-fold symmetry only holds for points 
in the image that remain within the image boundaries upon rotation by 
27/6. On the left of Figure 6.6 we show a larger scale view of the design, 
within a circular frame. This latter design clearly displays 6-fold rotational 
symmetry. 


For small magnitude z, we have f(z)~z~°. Consequently, the curved 
lattice contains an infinity of pre-images approaching the pole of 


multiplicity 6 at z = 0. The sizes of the pre-images decrease rapidly to 0 
as z —> 0, hence they become incapable of realization with the digital 
images that we are working with here. This explains the random looking 
colored pixels lying within a region interior to the six flower-like parts of 
image. Those flower-like parts lie across the unit-circle, where |z| ~ 1. 
Some of the curved lattice within the unit-circle is visible in the image on 
the right of Figure 6.6. 


FIGURE 6.6 


Left: Circular view of rosette design with 6-fold rotational symmetry plus curved lattice symmetry. Right: 
Curved lattice lines marked on rosette design with curved lattice symmetry. 


6.2.2.2 Animations 


We have also created animations involving our symmetric designs. We will just 
give one illustration of the ideas involved. The animation we describe can be 
found at this link: 


https://www.youtube.com/watch?v=PflvJPywXWs 


We produced this animation in the following way. First, we selected a unit-length 


complex number p = e”® for small positive 0. We then created a succession of 
designs using the mappings: 


C 
z—> (1+4%) +p" (i/4)2° + see =u+iv — [uw mod 1000] + é[v mod 1000] 


(6.7) 


forn = 0,1,2,...,99. This produced 100 separate designs that are displayed one 
after another to create the animation. The interesting thing about this animation is 
that it will appear to rotate in the positive angular direction on its outer part versus 
a negative angular rotation on its inner part. The reason for this is that p” has 
angle n@ for n = 0,1,2,...,99, which is a sequence of increasing angles in the 
counter-clockwise direction. The outer part of the design corresponds to values of 
z that have lengths larger than 1, and for larger values of |z| we have 


= & p” (1/4) 28 


so the outer part of the design successively rotates through the angles 
nO = arg(p"), n = 0,1,2,...,99. A similar argument shows that for z close to 
0, the design simultaneously rotates through the angles —n8. 


(1+) +p" (é/4)2° + 


6.2.3. Rotationally Symmetric Wallpaper Designs 


6.2.3.1 Four-fold and Two-fold Wallpaper Symmetry 


In Figure 6.7, we show a wallpaper design that is symmetric with respect to 4- 
fold symmetry and various reflection symmetries. For this design, we used the 
Reptile image. To create a function f having square lattice symmetry and four- 
fold rotational symmetry, we follow the procedure described by Farris [9]. First, 
we use as a basis the set of complex exponentials { Finn (z) = e27™(™2*"y) } for 
al m,ne€Z and z=a2+iyeEC. Any finite, or convergent, sum 
S° AmnEmn(z) is guaranteed to have the required translational symmetry. To 
obtain rotational symmetry, we employ group averaging. In this case, the group is 


the rotations given by powers of i = e?7'/4. The group average Win Of Emin 18 
3 


defined by Wrn(z) = + » oem ale™2), Any finite, or convergent, linear 
k=0 
combination f(z) “ss m,n(Z) is guaranteed to have both square lattice 


symmetry and ane rotational symmetry about the origin. 


FIGURE 6.7 


Left: 4-fold rotationally symmetric design, with additional symmetries. Unit cell on its right marked in 
green. Right: 2-fold rotationally symmetric design with no additional symmetries. 


The specific function fthat we employed is 


f(z) = [Wio(z) + Wo,-1(z)] + 0.5[Wi5(z) + W_s,-1(2)] 
+ 0.12[W_» 4(z) + W_42 (z)| a 0.057|W_¢ 3(z) safe W_3.6(z)] 


(6.8) 


In this equation, the functions {W,,,,(z)} are grouped in order to enforce 
reflectional symmetry through the x-axis. To see this, we first rewrite the terms of 
Wm,n to remove the powers of 7. The term En,» (iz) satisfies 


Emn(iz). = Emly + iz) 
= e!2(—my+na) 


= Binal?) 


Iterating this relation yields Eyy»(¢?7z) = E_m,—n(z) and Ein n(i?z) = E_nm(z) 
. Therefore, 


Winn(2) = 7 [Emn(2) + En,-m(z) + B—m,-n(2) + E-nm()] 
(6.9) 


Now, if we apply reflection through the x-axis to Em n(z), we obtain 


En(2) = Eqn (@= 19) 
= Em,-n(Z) 


Iterating this relation, we obtain BE, -m(Z) = Enm(Z), B—m,—-n(Z) = E-mp(Z), 
and B_nm(Z) = E_n,-m(z). Therefore, 


Winn (2) = 4 [Bm,-n(2) + Enm(z) + B-myn(2) + B-n,—m(2)] 
_ —n,—™m (z) 
Consequently, terms of the form 
a|Wm,n(2) + W_n,—m(2)| 


appearing in Equation (6.8), are symmetric with respect to reflection through the 
X-axis. 

The symmetry with respect to R, is apparent in the design in Figure 6.7, but 
there are other symmetries as well. For example, it has reflectional symmetry Ry 


about the y-axis. Since the y-axis is a rotation by 27/4 of the x-axis, the function 
f must be symmetric with respect to R,. More precisely, Ry equals the conjugation 


operation Pon) 4x P2r/4 in the symmetry group S, Other reflection symmetries 
follow from group operations, such as reflection through the line y= z and 


reflection through the line y = —z. 
This wallpaper design is symmetric with respect to translation by the two 


independent vectors 1 and jin C. The cells of the lattice generated by 1 and i are 
clearly evident. To the right of the design, we show a unit cell, the contents of 
which generate the entire design via translations by m-1+7-%, for m,n € Z. 
One interesting feature of the design is that the center of the unit cell is a point of 
4-fold rotational symmetry for the design. This can be proved as follows. 
Assuming this unit cell has its lower left corner at the origin, its center is + + + 1 
. The following mappings 


oe oj, Lae co ee Oe be fe Dt 
Fhe hg ep pe ee op a oe 


imply that 71° por/4 preserves 5 + +i and rotates the unit cell by 27/4. 
Consequently, ++ si is a center of 4-fold rotational symmetry. A similar 


argument shows that the midpoint of the top side of the unit cell is a center of 2- 


fold rotational symmetry. Hence, by four-fold rotation, the midpoints of each side 
of the unit cell are centers of 2-fold symmetry. 

Two-fold rotationally symmetric wallpaper patterns can be generated in 
multiple ways. One, rather elementary, way is to just stretch a 4-fold pattern to 
create a rectangular lattice. The rectangular lattice for a 2-fold design, that is not 
4-fold, is generated by the basis vectors 1 and ri, where r > 0 and r £ 1. Such a 
design can be generated from a 4-fold design by simply stretching or shrinking 
the 4-fold design in the vertical direction. With the digital images we are creating, 
that can be done by even the most rudimentary image processing programs. A 
second method would be to use a group average approach, as we did with the 4- 
fold case. For instance, we can use group averages 


ae ae me ? + Emn(—z)). A finite, or convergent, linear combination 


=> AmnWmn(z) will then have both 2-fold rotational and square lattice 
ar An ak is shown on the right of Figure 6.7. The function f(z) that 
we used for this design is 


f(z) = Wi 0(z) a Wo-1 (2) ae 0.5W15(z) ++ 0.1¢W_24(z) = 0.051W_¢ 3(z) 
(6.10) 


Unlike the design on the left of Figure 6.7, this design has no additional 
symmetries. This corresponds to the absence of grouping of related terms in 
Equation (6.10), of the kind that we have in Equation (6.8). 


6.2.3.2 Three-Fold Wallpaper Symmetry 


On the top left of Figure 6.8, we show a wallpaper design that is symmetric with 
respect to 3-fold symmetry. Next to the design is an image of a unit cell for the 
lattice of the design, a rhombus with sides constructed from the complex numbers 
1 and w = e’®"/3. The rhombic lattice for translational symmetry is defined by the 
vectors m + nw, for all m,n € Z and w = e”*/, Every complex number z can 
be expanded uniquely as z=u-+vw for unique u,v € R. The functions, 
Emn(z) for m,n € Z, are defined by Emn(z) = Emn(u + vw) = ei2n(mutny) 
for all u,v € IR. These functions are periodic over the cells of the rhombic lattice 
generated by 1 and w. The group averages, {Wi}, for the group of 3-fold 
rotations about the origin, are defined by 


Wm n(Z) = $ | Em,n(2) + Emn({wz) + Emn(w*z)| 


(6.11) 


FIGURE 6.8 


Top Left: Wallpaper design with 3-fold rotational symmetry. Top Right: a rhombic unit cell, marked in 
green. Bottom: Wallpaper design with 6-fold rotational symmetry and additional reflection symmetries. 


This definition of W,,,,(z) ensures that it has 3-fold rotational symmetry. To 
verify that it has translational symmetry, we need to further examine the terms 
Emmn(wz) and Em »(w?z). Since z=w satisfies the factored equation 
0 = (z2 —1) = (z—1)(2? +z +1), it follows that w? = —1 — w. Therefore, 


Emn(uw + vw?) 
Emn(—v + (u — v)w) 
e22(—mv+n(u—v)) 


= En,—m-—n (z) 


= e22n(nut+(—m—n)v) 


Thus, Emn(wz) = En—m-—n(z). Iterating this relation, we obtain 
Emn(w?z) = E_m_nm(Z). These relations show that 


Wan (z) 7 z Borner (2) a Eom (z) i hopenin (z)| 
(6.12) 


Hence, W,,,7 is a linear combination of the basis functions {En}, so it also 
enjoys translational symmetry over the rhombic lattice. Consequently any 
function f defined by a finite, or convergent, sum S° AmnWmn(z) has both 3- 
m,n 

fold rotational symmetry and rhombic lattice symmetry. Furthermore, because of 
the group operation En - Ej. = Em+jn+% for the basis functions {Emn}, we 
can also include products of the form W,,,, - W;,, in the terms for f Specifically, 
the function we used to create the design on the top left of Figure 6.8 is 


f(z) = 2Wo3(z) - Wr a(z) + Wi0(2) 


It has 3-fold rotational symmetry because each of the factors, W23 and Wj 4, and 
the term Wj have that symmetry. Moreover, it has translational symmetry over 
the lattice because fis a linear combination of the basis functions { En,» }- 


Remark 6.2.1. In this example, we were indeed fortunate that a rotation by 27/3 
maps each basis function E,,,,, periodic over the lattice, into another such basis 
function. That fact allowed us to create a 3-fold rotationally symmetric wallpaper 
design. The Crystallographic Restriction Theorem below implies that this can not 
happen for most n-fold rotational symmetries. 


6.2.3.3 Six-fold wallpaper symmetry 


In Figure 6.8, we show a wallpaper design that is symmetric with respect to 6- 
fold symmetry and various reflection symmetries. Before we discuss the 
mathematics of creating this design, we will take a moment to comment on its 


artistic features. The design presents an appearance of interlocking circular 
regions. The larger regions have a central, three-armed cross that appears to be a 
center of three-fold rotational symmetry. Slightly smaller circular regions have 
dark blue, hexagonally shaped figures about their centers, which appear to have 6- 
fold rotational symmetry. If you relax your focus slightly, the whole design 
appears to float in the background as your attention rapidly shifts between these 
interlocking circles with different symmetries. These features slightly disguise the 
location of the rhombus shaped cells that form the lattice for the design. 

We used the Buoy image for the color map of this design. To create a function 
fhaving rhombic lattice symmetry and six-fold rotational symmetry, we note that 
6-fold rotational symmetry about the origin is equivalent to 3-fold rotational 
symmetry combined with 2-fold rotational symmetry. Two-fold rotational 


symmetry about the origin in C corresponds to the mapping p,; : z > —z. Since 
Emn(—2z) = E_m,-n(z), it follows that Wirn(—z) = W_m,—-n(z) for all 
m,n € Z. Consequently, if fis a finite, or convergent, sum of terms of the form 


@|Winn(z) + W—m,—n(z)| 


then fis symmetric with respect to R, and R»,/3 and thus symmetric with respect 
to Ro, /6- 

The function f that we used to create the design in Figure 6.8 is somewhat 
complicated. Suffice it to say that it had the following form: 


f(z) = [We 3(z) + W3,2(z)] + a[W_2_-3(z) + W_s,-2(z)] 
- [Wi 5 (z) - W51 (z)| St b[W_1-5 (z) ap W_5-1 (z)| 
a c|W3.4 (z) a Wa3 (z)| ae c|[W_3 4 (z) =F W_4-3(z)] 


(6.13) 


for certain complex constants, a, b, and c. The terms for f(z) can be regrouped so 
that fis a linear combination of terms of the form Wrn(z) + W_m,—n(z). Hence, 
fhas 6-fold rotational symmetry. However, the terms for f were grouped in the 
pairs shown in Equation (6.13) in order to enforce reflection symmetry R, about 
the x-axis. To see that this symmetry holds, we examine the effect of Rz : z > Z 
on Emin: 


Emn(Z) = Emn(ut vo) 
= En, »(u + v(—-1—w)) [since @=w?| 
3 e22m(mu+(—m—n)v) 


a Em,—m-—n (z) 
Thus, Emn(Z) = Em,—m—n(2). Iterating this relation, we obtain 
En,-m-n(2Z) = Enm(z) 9 and = En—nm(Z) = B-m-nn(z) 


Combining these relations with Equation (6.12), we obtain Winn (Z) = Wnm(Z). 
Consequently, terms of the form 


alWinn (z) + Wrim (z)| 


are symmetric with respect to reflection through the x-axis. Since fis a finite sum 
of such terms, it has this symmetry, too. Our design has other symmetries as well. 
For example, since pz Ry : z — —Z, the function fis mirror symmetric through 
the y-axis. 
At the beginning of our discussion of this 6-fold symmetric design, we 
mentioned circles having centers of 3-fold rotational symmetry within the 
1 


rhombic cells of the design's lattice. The point 4 + 3 w within the rhombic unit 


cell is a center of 3-fold symmetry. The following mappings 


“W TI 
$+ fu3 t4tu3 dete 

imply that 7) ° p27/3 preserves + + $ w and rotates the rhombic unit cell by 27/3. 

Consequently, + + s w is a center of 3-fold rotational symmetry. Furthermore, 


Tl+w 


eis Oe eT to 
gt ZW > Gg gW—> et gw 


implies that 714, P2n/3 preserves $ + 2 w, and so it is a second point of 3-fold 
symmetry within the rhombic unit cell. 


6.2.3.4 The crystallographic restriction 


We have shown that wallpaper designs can be generated with n-fold rotational 
symmetry when n = 2,3,4, and 6. In fact, these are the only possible n-fold 
rotationally symmetric wallpaper designs. 


Theorem 6.2.2 (The Crystallographic Restriction) An n-fold rotationally 
symmetric wallpaper design is only possible if n = 2, 3, 4, or 6. 


Proof. In the standard basis for R? = C, the n-fold rotation p», /n has matrix form 


anu 7 


sin(27/n) cos(27/n) 


The trace of this matrix is 2 cos(27/n). However, the trace is invariant under 
change of basis. For the basis that generates the lattice of cells for the wallpaper 
design, the trace of the matrix for ,/, must be an integer. Hence, 
2 cos(27/n) = k for some k € Z. Therefore, we have 


cos(27/n) =k/2, keEZ 
(6.14) 


The only possible integers k for which Equation (6.14) can hold are 
k = 0, +1, +2, which yield n = 2, 3,4, and 6. Oo 


There is an interesting analysis of this restriction when n = 5 in [10]. Some 
Matlab® programs for creating designs when n = 5 are in [28, p. 98]. These 
designs are related to quasicrystals[25, 28]. 


6.2.4 Summary 


We have shown a number of symmetric designs generated by the application of 
complex analysis to the geometry of the Euclidean plane. The mathematics we 
have used is widely employed in crystallography [18, 25]. Many more designs, 
and a more thorough treatment including the relation to crystallography, can be 
found in the book by Farris [9]. In the next section, we describe symmetric 
designs that use properties of complex analysis in hyperbolic geometry. 


ee 
6.3 Designs in the Hyperbolic Upper Half-Plane 


We have also created designs using the symmetries in the geometry of the 
hyperbolic upper half-plane. Some of the designs we have created are shown in 
Figure 6.9. The symmetries in these designs are much different than those we 
discussed above for the Euclidean plane. The design at the top of Figure 6.9 is 


entitled Blugold Fireworks. It was exhibited as part of the 2018 Mathematical Art 
Exhibition held in San Diego [13]. The other two designs are more recent 
creations. One quite interesting feature of these designs, from a mathematical 
perspective, is that they display some of the principal geometric objects in the 
hyperbolic upper half-plane. We will describe what these principal objects are, 
and how the designs are constructed. But in order to do so, we first recount the 
basic mathematics underlying the geometry of the hyperbolic upper half-plane. 
References for additional details are [5, 9, 16, 20, 28]. 


FIGURE 6.9 


Top: Blugold Fireworks design, using Buoy as color map. Middle and Bottom: Two designs using Reptile. 
6.3.1 Geometry of the Hyperbolic Upper Half-Plane 
The hyperbolic upper half-plane, H, is the subset of C defined as follows: 


H={x+iy|z2eR,y>O0} 


(6.15) 


with differential metric 


ds = Varrav 
y 
(6.16) 


With this metric, the length f(y) of a smooth curve y(t) = x(t) + iy(t), 
a <t <b, is defined as 


oe is fo'(t)? + y(t)? re [2t(a)| 
ie) / att = / dt 


Im z(t) 


This metric is related to the metric dsg+ for the Euclidean plane, restricted to 
Et = {(z,y)|a € R,y > 0}, defined by 


dsp+ = \/ dx? + dy? 


(6.17) 


In other words, ds = dsg+/y. This relation is crucial to verifying a number of 


important facts about the geometry of H. We begin by discussing the isometries 
of H. 


6.3.1.1 Isometries of H 


The isometries of H are mappings f : Hl — H that preserve the differential metric 
ds. We will show in the next theorem that the set 


Sa = { f(z) = sett : a,b,c,d € Rwithad — be = 1} 


(6.18) 


contains all the holomorphic isometries of H. 
There is also a notion of area in H. The area of a region YW will be denoted by 
A(@%). The area differential dA in His given by 


dA = a dx dy 
(6.19) 


and so we compute A(Y) by 


(6.20) 


for a suitable region Y C H. By a suitable region, we mean any region Y Cc H 
for which the integral in (6.20) is defined, say, as a Riemann integral. The area 
differential dA makes sense, by a dimensional argument, when viewed as 
dA = dAg:+/y’ and noting that the length differential satisfies ds = dsp+/y. It 
also follows from basic facts of Riemannian geometry: 


al 


ij=l = 
(differential metric squared) (differential volume element) 
; 1/y2 0 
as shown in [2, p. 188, 241]. In this case, (g:,;) = 0 Uy? , and we obtain 
y 


ds and dA as defined in (6.16) and (6.19). We will show that isometries in Sj, also 


preserve area in H. 


Theorem 6.3.1. The set Sj;, defined in (6.18) contains all the holomorphic 


isometries of H. In addition to preserving the metric differential ds, these 
isometries also preserve the area differential dA. The isometries in S,, are also 


described by 


Su = { f(z) = set : a,b,c,d € Rwithad — bc > o} 


(6.21) 


Proof. For f(z) = +5 in S,, we have 


(ad—bc)(z—zZ) 
|cz+d|? 


2iIm f(z) = 


1 : 
= Jerae 22m z 


and therefore Im f(z) = Imz/ |cz + d|?. Consequently, f(z) € H if and only if 
z € H. (Note: cz + d = 0 is only possible when z = —d/c € R and such z are 
not in H.) 

Now, for f(z) = u + iv with u € R and v > 0, we have v = y/|cz + d|?. We 
also have 


du? + dv? = |J|(dx? + dy’) 

= |(24)’ + (34)"] (dx? + dy’) 
Ne )|? (da? + dy”) 
(dx? + dy’) 


7 eae 


where we made use of the Cauchy-Riemann equations to simplify the Jacobian 
|J| for the change of variables in the first line. Consequently, we obtain 


du2+dv2 ___ |ez+d|* 
v2 — gP ear (dx + dy ) 
= dz?+dy? 
x 


and that shows that f(z) is an isometry of H. Moreover, in our calculations we 


computed f'(z) for z € H, so fis holomorphic on H. 
To prove preservation of dA, we calculate as above: 


du dv = pba 


dx dy 


_ = ear 


and dudv/v? = dxdy/y’ follows just as above. Thus, the isometry f also 
preserves the area differential dA. 
Now, suppose that f(z) = ante with ad — bc > 0. Let r? = ad — bc. Then, 


— r? alz+b! 
f(z) ~ 72 clz+d! 
a'z+b! 
c!z+d! 


with a’d' — b'c' = 1. Therefore, f(z) is in Sy, as defined in (6.18). Since the 
reverse inclusion obviously holds, it follows that Sj) is described by both (6.18) 


and (6.21). 

Finally, by [4, Theorem 5], all holomorphic mappings from the disc 
D = {z: |z| < 1} to itself have the form F(z) = t(z — c)/(1 — éz), for some 
t,c € C with |t} = 1 and |c| < 1. Conjugating each F with the conformal map 


g:D—H given by g(z) = at , we obtain all the functions f = g° F’g™* that 


belong to Sj, We omit the details for verifying this last statement, because we 


will not be using the fact that S;, consists of all the holomorphic isometries of H. 
Complete details are in [26, Theorem 2.4, p. 222]. Oo 


Remark 6.3.2. Theorem 6.3.1 deals with the holomorphic isometries of H. 
There are other isometries. For example, the function f(x + iy) = —ax + ty is an 


isometry, since it clearly preserves ds. However, it is not holomorphic on HH due to 
its failure to satisfy the Cauchy-Riemann equations. 


Theorem 6.3.1 tells us that these sets of mappings are all isometries: 


1. &@,:z-—> pz, for p>0. In E*, this mapping would be a similarity 
transformation when p # 1, not an isometry. In H, however, this mapping 
is an isometry for all p > 0. 

2. Ty: 2—>2z+ 4, for u € R. Thus, all horizontally oriented translations are 
isometries of H. 


3, Grr Z- ae for r > 0. This operation is called inversion through the 
circle of radius r, center 0 in C. However, it is also inversion through the 
upper semicircle in H defined by S$} = {z € H : |z| =r}. In other words, 
55 = {x +iy: 2+y=ry> oO}. In subsequent work, we shall also 
deal with upper semicircles of radius r and center u € IR, which we denote 
by $7. Note that $7 = 7,,(5%) and S} = .4,(S}). 


These special isometries generate all the isometries in S,, through composition. In 
fact, if c ~ 0, then by long division we obtain 


az+b a4 b—da/c 
cztd — ¢ cz+d 
— & oes ede: 
— ¢ a ce? 2z+d/c 


hence f = ase M102 °F, Tdies While if c = 0, then 


aztb a ob 
= eer" 


Since c = 0, ad — bc = 1 reduces to ad = 1, and we have 


f(z) =a°(z + b/a) 
Thus, f = M2 °Tp/a- 


6.3.1.2 Geodesics 


A geodesic in H, connecting two points z, and Z, is a (piecewise) smooth curve 
y(t) = a(t) + iy(t) fora < t < b that satisfies y(a) = 21, y(b) = z2, and which 
has minimum length ¢(7). We shall now prove that, in E*, these geodesics lie 
along vertical rays or semicircles. 


Theorem 6.3.3. The geodesics in H lie along the following two types of curves in 
BR (69) vertical rays Ry emanating from R: 
Ry, = {utity: y> 0, fixedu € R}, or (2) open semicircles S’ centered on R: 
c= {e+ iy : (x —u)’ +y=r?,y> 0, fixedu € R,r > 0}. 

Proof. First, we consider z; = zy; and z2 = 72y2, choosing subscripts so that 
Y2 > y; > 0. For a smooth curve y(t) = x(t) + zy(t) satisfying y(a) = y, and 
7(b) = ya, we have 


Moreover, this lower bound of In(y2/yi) is realized for 
y(t) = [yi + (y2 — y1)t|i for 0 < t < 1. Therefore, this function y is a geodesic 


in H, and clearly it lies on the ray Zp. The minimum property also extends to the 
class of all continuous, piecewise smooth curves, by splitting integrals over [/a,b/ 
into finite sums of integrals. Since the horizontal translation 7, is an isometry, 


conjugation with t,, implies that geodesics also lie along each vertical ray Z,,. 


Second, we consider two points z; # zz on the open semicircle $1, having 
arg(z2) = 02 > 0; = arg(z1). The isometry f(z) = (z+ 1)/(—z+1) maps Sj 
to Zo, with iys = f(z), iyi = f(z), and ye >yi. Given a geodesic 
y(t) = a[(y1 + (y2 — y1)t] for O<t <1, connecting iy, and iyz on Zo, we 
apply the isometry f~! to obtain f~'*y as a geodesic on S$} connecting z, and 


Z. Thus, Sj contains geodesics in H. Since the isometry 4, maps Sj to Sf it 
follows that Sj contains geodesics in H. Finally, since the isometry t, maps S9 to 


S7, it follows that S7 contains geodesics in H. i 


Remark 6.3.4. The distance d(z,, z,) between two points z1, z2 € H is defined to 
be the length of a geodesic that connects z, and z,. For example, we found above 
that d(x +iy1,2+2y2) =In(y2/y1) for y2>y,>0. In general, for 
z+iyi,c+iy2¢€H, we have d(x+iy1,z+iy2) =|Mm(y2/yi)|. It is 
important to note that |In(y2/y,)|— oo if either y; ~0 or yo 0. 
Consequently, the real line R is a line at infinity for all points in H. 

There is a distance formula for all z,w € Hl, given by 


d(z,w) = In eh ul 
’ |z—w|—|z—w| 


(6.22) 


but we will not need it. Interested readers will find a proof of (6.22) in Katok [16, 
Theorem 1.2.6, p. 6]. 


For simplicity, in the rest of the paper, we shall refer to rays of type #, and 
open semicircles of type S” as geodesics. Strictly speaking, they contain 
geodesics, but there is little chance of confusion and our language is more 


straightforward if we simply call them geodesics as well. These geodesics in H 
can be interpreted as a model for the undefined term Jines referred to in postulates 
of geometry. In fact, these geodesics do satisfy the first four of Euclid's postulates. 
However, they violate the notion of Euclidean parallelism. For example, in the 
image at the top of Figure 6.10, the two geodesics on the left intersect at a point. 


Yet, they fail to intersect the vertical geodesic on the right of the image. This 
situation violates the uniqueness of a parallel line, through a point not on a line, 
required in Euclidean geometry. 

Returning to the artworks in Figure 6.9, it is interesting that parts of these 
designs correspond to geodesics. On the top of Figure 6.10 we have shown that 
geodesics of both types, S7 and Z,, are evident within the design shown at the 
bottom of Figure 6.9. Parts of the other two designs also correspond to these types 
of geodesics. For instance, on the bottom right image in Figure 6.10 we have 
shown how a part of the middle design in Figure 6.9 corresponds to both types of 


geodesics in H. 


FIGURE 6.10 


Top: Four geodesics, drawn in yellow over one of our hyperbolic designs. The two intersecting geodesics 
on the left are of types Z,, and S’. The two disjoint geodesics on the right are also of types Z, and S’. 
Bottom Left: Portion of the same hyperbolic design that contains horocycles. Bottom Middle: Yellow 
circles indicating some of these horocycles. These horocycles are orthogonal to geodesics. Bottom Right: 
Yellow semicircle and green vertical line indicating geodesics on the middle design in Figure 6.9. 


The images at the bottom of Figure 6.10 also contain geometric objects 


related to geodesics in H. These objects are circles that are tangent to K at one 


point and have all other points lying in H. To be specific, for u € R and r > 0, a 
horocycle H,.(u) is defined by 


H,(u) = {a +iy :(a—uy+ly ry =r?,y > oh 


so in Et it is a circle with center (u,r) € R?, and radius r, but omitting the point 
(u,0) on the x-axis. These horocycles are not geodesics. However, we will now 


discuss how the family of all horocycles are orthogonal curves in H for the family 
of all geodesics. 


6.3.1.3 Angles and Conformality in H, Horocycles and Geodesics 


An angle in H is defined to be an angle between tangent vectors of two curves 
meeting at a point. The following theorem shows that these angles are the same in 


both H and E*. 


Theorem 6.3.5. Let 0); and 0g+ be the angles between two curves at some 


intersection point in H and E”, respectively. Then, 04 = Og+. 


vi 


Proof. We can write the infinitesimal quadratic form ds? = (dx? + dy*)/y? as 


\[da, dy] 


where ([dz, dy], |dx, dy]),+ stands for the standard inner product of the vector of 
differentials /dx,dy] with itself, and ||[dz, dy]||?, is our notation for ds” thought of 
as a quadratic form of the vector /dx,dy]. The inner product ([dz, dy], |d%, dG]) 1 
corresponding to this quadratic form is then 


(|dz, dy], |dz, dy)) 1 — ee 


Consequently, cos Oy satisfies 


([dx dy), [dz,d9]) 5s 

cos@y = — eee eee 

Hl I[d2,dy] la || [4,49] | 
dx d&+dy dg 


~ /dx?+dy? ./d#+dy? 


= cos Or+ 


vi 


Thus, we must have 04 = 65. oO 


Since angles in H and E* always correspond to angles between tangent 
vectors, we have proved that angles in the two geometries are always the same. 
The two geometries are said to be conformal. 


Corollary 6.3.6. The isometries in S}, preserve angles in H. 

Proof. If f(z) =(az+b)/(cz+d)€Su, then f’(z) =1/|cez+d|? £0. 
Therefore, fis a conformal mapping on E* by [4, Theorem 3]. Hence, Theorem 
6.3.5 above implies that fis a conformal mapping on H. Oo 


This corollary can also be proved using the identity 
2(v,w) = |v + vl? — [Ivil? — [lwil? 


relating inner products and quadratic forms, and the fact that an isometry 
preserves the quadratic form ds?. However, our proof highlights the relation 
between the geometries of E* and H. 


We now return to the concept of horocycles in H, and how they are illustrated 
in the designs shown in Figure 6.9. The simplest type of horocycles are the sets of 
form, {t+ iv : t € R}, parameterized by varying iv with v > 0. These sets are 


horizontal lines in E*, but in H/ they are not geodesics. Each geodesic ray #,, for 
u € R, lies orthogonal in IE* at each of its points to one of these horizontal lines, 


and therefore each geodesic ray &,, also lies orthogonal in H at each of its points 
to one of these horocycles {t + iv,t € R}. 

The second type of horocycles are those that lie orthogonal to points of open 
semicircle geodesics. The isometry .%,(z) = —r*/z maps the geodesic ray Zp to 
itself (with ri held fixed), and maps the geodesic ray Z_, to the open semicircular 


geodesic S. us 


r/2" The horocycles for Z_,, expressed as {t + ryt : t € R} for each 


y > 0, are mapped by -Y, to sets of the form {wv eH: lw — oT il’ = (=) }. 
which are circles in E* except for the one point 0 + 02 ¢ HI. By Corollary 6.3.6, 
these horocycles are orthogonal to the open semicircular geodesic Ss. 5 at all of 
its points. They are circles in E* that are tangent to the point (0,0), and all their 
points excepting (0,0) lie in H. Conjugating with horizontal translation 7, for any 
fixed wu € R, we find that the horocycles for all open semicircular geodesics in H 
are circles in E* except for one point that is tangent to R. Since these horocycles 


are all tangent to K at the same point u, with all radii r > 0, it follows that each 
family of horocycles is also orthogonal to the geodesic ray #, at each of its 
points. On the bottom left and bottom middle of Figure 6.10, we illustrate a 
collection of such horocycles in one of our designs. The geodesics drawn on the 
design at the top of this figure are orthogonal at each of their points to such 
horocycles. On the bottom right of Figure 6.10, we show a part of the middle 
design in Figure 6.9 that exhibits both horocycles and geodesics. 


Creating designs with hyperbolic symmetry 

Our method for creating designs with hyperbolic symmetries is similar to our 
method for Euclidean symmetries. We symmetrize a given function fwith domain 
H. The symmetries will be a subgroup of Sj; We cannot use Sj, itself because the 
only functions on H, symmetric with respect to all the transformations in Sj, are 


constant functions. Following Farris [9], we will use the subgroup I’ known as the 
modular group. The modular group I is defined as 


r={f@= ieee : j,kym,n © Z,jn— mk = 1} 


mzt+n 


(6.23) 


) k 
Note that jn — mk is the determinant of the matrix (; ) of coefficients of 
mn 


f(z) = j2vk The set I is a group because composition of two members 


mzt+n 


f(z) = Bee. and g(z) = es r ~ satisfies 


° — (j' +km')z+(jk'+kn') 
(F°9)(2) = Garam oe) 


which corresponds to multiplication of the matrices of coefficients of fand g: 


j k 7 k! 7 97. ae km! ak! ae kn! 

m n)\m' vn’) \mj’+nm! mk! +nn' 
and we know that determinants of matrices respect multiplication. So the 
determinants of each of the matrices in the equation above satisfy 1-1=1, 


nz—k 


ree and therefore 


hence f°g is a member of I. Furthermore, f~+(z) = 
foe 


Remark 6.3.7. The group Sj, is isomorphic to a subgroup of the matrix factor 
group: 


PSL(2,R) = SL(2, R) / {Id, —Id} 


where SL(2, R) is the special linear group consisting of all 2 by 2 matrices over 
RK having determinant 1, and Id is the 2 by 2 identity matrix. The group 
PSL(2, R) is related to the projective geometry of all lines through the origin [5, 
p. 179]. It is called the projective special linear group over R. The group IT is 
isomorphic to PSL(2, Z) = SL(2, Z) / {Id, —Id}, where SL(2,Z) consists of 
matrices in SL(2, R) with integer coefficients. 


The special isometries, 4, t,, %,, mostly are not members of I due to the 


requirement that their coefficients belong to Z. In fact, the special isometries that 
belong to I’ are 


1. Translations: T” : z > z+n, for n © Z. The unit-translation T! will be 
written as just 7. These translations obey the group operation in I: 
LL = Pe" {or al mi. SZ, 


2. Inversion: Y : z > —1/z 


Compositions with these isometries are sufficient to generate all the isometries in 
Tr. 


Theorem 6.3.8. The unit translation T and inversion Y generate I. More 
precisely, if f © I’, then f can be written as some interlaced composition of Y 
with various translations T?: 


L 
f= (11 mms) TPo 


=1 


(6.24) 


Proof. Let f(z) = ** be an arbitrary function in I. First, suppose m = 0. 


mz+n 


Then f(z) = (j/n)z + (k/n) and we have jn = 1. Consequently, (j,) = (1, 1) 
or (—1,—1). Hence, either f(z) =z+k or f(z) =z—k, and so f =T* or 


f =T~*. Now, ifm + 0, we reduce to the first case as follows. We have, where 
LEZ, 


If(2)= BB, f(z) = ee) 


jztk mztn 


If |j| > |ml, the division algorithm gives 7 = mq +r with 0 < |r| < |m|. Hence 
Pts tz) = — Then apply ¥ to obtain YT “f(z) = LEER), with 
Ir| < |m|. Applying powers of T, followed by .Y, eventually results in a 
remainder r = 0 as coefficient of z in the denominator. That is, we arrive at 7’?° 


for some po € Z. Thus, we obtain 


L 
(11 sr) f=TP 
s=1 


Solving for f we obtain the result in Equation (6.24). Oo 


The symmetrization fg of a function f defined on H, can be done as follows 


fs(z) = >) f(9(2)) 


gel 


(6.25) 


Since I is a group, the function fg is guaranteed to satisfy the symmetry condition 


f(g(z)) = f(z) for all g € I. In practice, of course, we can typically only create 
partial sums of the infinite series for fg. Nevertheless, as shown above, our 


designs using such partial sums display many important features of the geometry 


of H. 
To create designs by domain coloring of fg(z), we need to express the series 
defining f; in a more convenient form. To do that, we observe that the condition 


gn — mk = 1 can be rewritten as 
nj+(-—m)k=1 
(6.26) 


Equation (6.26) is a famous one from Number Theory. It is equivalent to the 
integers j and k being relatively prime, i.e., their greatest common divisor is 1, 


which we write as gcd(j,k) = 1. The numbers n and —m are called Bézout 
coefficients for j and k, and they ensure that Equation (6.26) holds. Because 


) ok 
det (? ) = | 
mn 
will hold if j and k are perturbed by 7 = £m and k = én for @ € Z, these Bézout 
coefficients determine all transformations g having the form 


g(z) = + SE 


mz+n 
where (n,—m) is a single pair of Bézout coefficients for (j,k). Therefore, we 
will assume that the initial function fhas period 1 in the x-variable, and write fg 
as 


fs(z) = s f( 25) 


ged(j,k)=1 
(6.27) 


To efficiently calculate the series in Equation (6.27), we use a recursive, tree- 
based method for computing relatively prime pairs of positive integers and 
associated pairs of Bézout coefficients. Randall [22] proves that pairs of relatively 
prime positive integers can be computed using F(j,k) = (27+k,7) in the 
following recursive formulas: 


FG,k) = (27 +4k,3) 
F(k,j) = (2k+3,k) 
ae —k) = (23 -_ k, j) 


starting from either (2, 1) or (3, 1) as initial pair. This recursive calculation 
generates two distinct trinary trees with roots (2,1) and (3,1), as illustrated in 
Figure 6.11. We have found that there is a similar recursive computation for 
finding associated Bézout coefficients that works for both of the nodes (j,k) in 
these trinary trees. In [14], we show that using G(u, v) = (v,u — 2v) and the 
recursive equations 


(3,2) —— (8,3) (—1,2) — (2,-5 (5,3) — (13,5) 


(2,1) — (5,2) —— (12,5) (0,1) — (1, -2) — (2.5 (3,1) — (7,3) — (17.7) 


(4,1) — (9,4) (0,1) — (1,-2 (5,1) — (11,5) 


FIGURE 6.11 


Left: Trinary tree of relatively prime positive integers generated by (2, 1). Right: Trinary tree of relatively 
prime positive integers generated by (3, 1). Middle: Trinary tree of Bézout coefficients generated by (0, 1), 
corresponding to both of the other trees. All trees are shown to a depth of 2. 


starting from (u,v) = (0,1), generates Bézout coefficients (u,v) satisfying 
uj +vk =1 for each node (j,k) in both trees of relatively prime positive 
integers. See the tree in the middle of Figure 6.11. From these Bézout 
coefficients, and their associated pairs (j,k) of relatively prime positive integers, 


we obtain transformations g(z) = “* eI. 


mzt+n 


Based on this recursive, tree-based organization of elements of we compute 


symmetrized designs by the following procedure: 


if 


Hyperbolic Symmetrized Design Procedure 


Start with a function f(z) having period 1 in x. It corresponds to 
(j,k) = (1,0) and (m,n) = (0,1). Then add f(—1/z), corresponding to 
(j,k) =(0,-1), and (m,n)=(1,0). For this step, we have 
f(z) + f(-1/z). 

Run through the trinary trees of relatively prime integers (j,k), starting at 
the roots (2,1) and (3,1). For each pair (j,k), and associated Bézout 
coefficients (m, —n) at the corresponding node in the Bézout tree, add the 


terms f ( rane ) and f ( aUee ) to the terms already summed. 


Z+n mz—-n 


3. After adding a large number of terms—we typically used about 400 terms 
—create a symmetrized design using the Domain Coloring Procedure in 
Figure 6.2. 


The designs in Figure 6.9 were all created using this method, starting with various 
functions f For example, the design in the middle of Figure 6.9 was created using 


f(a + iy) = 2iycos(2rx) + 2ysin(27y/3) 


An animation illustrating the steps in the method above, in the construction of the 
Blugold Fireworks design, can be found at the link given in [15]. 


6.3.1.4 Rotational Symmetry and Tessellation of H 


The designs we have created relate to a number of other additional aspects of the 


geometry of H. For example, on the left of Figure 6.12 we show a clip from the 
third design in Figure 6.9. Overlayed on this clip is a circle surrounding a point of 


3-fold rotational symmetry in H#. It is important to note that this 3-fold rotational 
symmetry exists within H and not within E*. To see that we do have 3-fold 


symmetry in H, in contrast to what we are used to seeing with Euclidean 
geometry, we need to discuss some further ideas from hyperbolic geometry. We 
first begin by describing the significance of the yellow circle on the left of Figure 
6.12. Its center in E* is marked by a blue dot. This circle is the locus of points 
that are a fixed distance p from the yellow dot in H. To be precise, we have the 
following theorem. 


Theorem 6.3.9. For fixed p > 0, the locus of points C,(x + iy) that are distance 
p in H from a fixed point x+y is equal to the circle in E* with center 
x + tycosh p and radius sinh p. (See Figure 6.12.) 

Proof. Begin by supposing that the fixed point is 7. Using the distance formula 
d(iyi, ty2) = | In(y2/y1)|, the two points te? and ie? are both distance p from i 
in H. So, ie’, ie” € C,,(z). The midpoint on the axis in E* between these two 
points is 2 cosh p, and it is Euclidean distance sinh p from both points. Now, map 
C(t) in H to C,(0) in the unit disc D, using the isometry 
f(z) = (tz +1)/(z¢+17) from H to D described below in Section 6.3.1.5. As 


discussed in that same section, the metric differential dsp has rotational 
invariance about 0, and therefore C’,(0) is a Euclidean circle about 0 (although its 


radius is not equal to p). Then map C’,(0) back to C’,(z) in H using the isometry 
f-\(2) = (2+ 7%) / (iz +71) from D to H. Because f~1(z) is a linear fractional 
transformation, it maps Euclidean circles to Euclidean circles, hence C’,(2) is a 
circle in E*. Since we found its center and radius must be 7 cosh p and sinh p, we 
have proved the result for fixed point i. Conjugating with T,.Z,, we get the result 
for fixed point x + ty. Oo 


FIGURE 6.12 


Left: Illustration of circular region in H with center at point of 3-fold rotational symmetry in H. Right: 


Hyperbolic circle of radius p centered at the point z + zy € H, and its Euclidean center and radius in Et, 


The yellow dot, located within a triangular region with curved edges on the 
left of Figure 6.12, is a center for a 3-fold rotation in H. It is located at 
1/2 + (/3/6)%. Before we show that it is a center for a 3-fold rotation in H, we 
show that the point z; = 1/2 + (/3/2); is also a center for a 3-fold rotation in 
H. This point z, is marked by a green dot on the graph on the top of Figure 6.13. 
It is related to a tessellation of H that we will discuss shortly. For now, observe 
that it lies at a vertex of a region labeled T’.Y in the tessellation. We will show 
that z, is a fixed point for TY, and that TY has order 3 in the group I’. We have 


_ =i = 3/9); 


sO Z, 1s a fixed point for T’.Y. Moreover, we have 


T#(z) = =, (T.Z)"(z) = —, (T.Z)*(z) =z 


which shows that TY has order 3 inT. 


We now turn to z2 = 1/2 + (W/3/6)i. We observe that z2 = .~T~?z,. But, 
IT ~* is an isometry. Hence we can apply the following Lemma: 


Lemma 6.3.10. Suppose z is a fixed point for g € Sy, and that g has finite order 
k. If f € Sy, then f° g° f~* has fixed point f(z) and order k. 


Proof. We find that (f° g° f~+)° f(z) = (f°9)(z) = f(z) so f(z) is a fixed point. 
Moreover, (fg? fy = f°g)° f—' for any integer 7 > 0. When j = k, we have 
gt =Id, so (f°g’ f-!)* = f° f-! =Id. Also, when j <k, if f° 97° f-! = Id, 
then we would have g’ = f~!° f =Id and that would contradict k being the 
order of g. Consequently, f° g° f~! has order k. o 


Applying the Lemma, we see that 1/2 + (/3/6)i = YT ~*z, is a fixed point 
forh = T-2°T ¥° (IT?) * and Ahas order 3. 
Returning to the left of Figure 6.12, the significance of the yellow dot and the 


circle enclosing it can now be explained in terms of rotation in H. Since the 
yellow dot corresponds to the fixed point z, for the isometry A of order 3, it is 


analogous to a 3-fold rotation in E*. In fact, for points sufficiently close to z,, the 


metric differential ds = dsg+/y is approximately equal to a multiple of dsm. 


Consequently, the isometry h is acting like a 3-fold rotation in E* in the limit of 
approaching z,. The enlargement of parts of the design as one rotates towards the 


vertical corresponds to what we see in E*. By Theorem 6.3.1, we know that area 
is preserved by isometries. Hence, in H, the upper arm of the figure along the 
vertical direction, has exactly the same area as each of the two lower arms of the 
figure (extending out from the edges of the curved triangle). 

There are also many other centers of 3-fold rotation in H that are illustrated in 
this design. If we let z, = IT* 2; for k = —3, —4, —5,..., then Lemma 6.3.10 
implies that we have centers of 3-fold rotations at each z,. Since each point 
T*z, = z1 +k lies along a horizontal horocycle in Et, applying the inversion % 
maps them to a circular horocycle in E*. We can see some of these points z, in 
the image on the bottom right of Figure 6.13. They are lying above the blue curve, 
extending downwards towards the bottom left corner, which is a slightly lower 


horocycle belonging to the same family of horocycles tangent to 0 in R. Finally, 
let z, = T.YT*z, for k = 2,3,4,.... This produces another collection of centers 
of 3-fold rotations that are on a second horocycle that moves away to the right of 


Z, towards | in R. Some of these centers are visible in the figure as well. For all 
of these centers, we observe repetitions of the curved triangle containing z, but at 


smaller scale. They are only smaller scale in E*. In H, isometries preserve area, 
so these curved triangles all have the same area in Has the one containing Z), 1.e., 


they are of the same scale in H. 
In addition to 3-fold centers, there are numerous centers of 2-fold rotational 


symmetries in H. The point z = 7 is a fixed point for 4 which has order 2. It is 
shown as a red dot in the tessellation at the top of Figure 6.13. Applying 
isometries to 7 we obtain sequences of 2-fold centers lying along horocycles. 
These centers of symmetry are plotted as red dots on the image shown on the 
right of Figure 6.13. 

We have discussed the relation between our design and the tessellation shown 
at the top of Figure 6.13. We shall now discuss this tessellation in more detail. 
Equation (6.24) shows how any f € I can be written in terms of powers of T 
interlaced with -Y. Starting from a fundamental domain, indicated by the shaded 
region ¥ in the figure, and applying compositions of 4% with powers of T 


generates this tessellation of H. In fact, the construction of the tessellation 
reproduces all the possible combinations of powers of T interlaced with Y in 
Equation (6.24). We now make all of these ideas precise with the following 
theorem. 


FIGURE 6.13 


Top: Tessellation of H generated by Tand -¥. Regions are labeled by transformations that produce them 
from shaded region ¥. Bottom Left: Location of center of 3-fold symmetry (yellow dot). This center lies at 
the intersection of three geodesics, shown in blue and orange in this image and in the tessellation above it. 
Bottom Right: Centers of 3-fold (yellow dots) and 2-fold (red dots) symmetries located along horocycles. 


Theorem 6.3.11. The modular group generates a tessellation of H via 


H= U f(F) 


fel 


(6.28) 


where F = {z EH: |z| >1,|Rez| < 1/2}, and every pair of regions f(F) 
and 9(¥) for f # g have disjoint interiors. 


Proof. First, we show that H = oe f(F). Let z be an arbitrary element in H. We 
€ 


will show that there is a g(z) = ee € I for which w = g(z) € F. Then we 


will have f(w) = z for f = g~*, which will establish the decomposition of F in 
Equation (6.28). Our main tool is 


Im 9(z) = Imz/|mz + nJ? 
(6.29) 


which was shown for all isometries at the beginning of the proof of Theorem 
6.3.1. Since there are only finitely many m,n € Z for which |mz+n| < 1, it 
follows from (6.29) that there are only finitely many g€J’' for which 
Im g(z) > Imz. Therefore, we can choose a g€I' for which Img(z) is 
maximal. If |Re g(z)| > 1/2, then we may compose g with some power of T so 
that |Reg(z)| < 1/2. Therefore, without loss of generality, we assume that 
[Re g(z)| < 1/2. Then we must have |g(z)| > 1, because if |g(z)| < 1 we would 
have 0<Img(z)<1, hence Im|%g(z)] = Im[ zy] >Imog(z) and _ that 
contradicts the maximality of Im g(z). Thus, |Re g(z)| < 1/2 and |g(z)| > 1, and 
sow = g(z) € F. 

Second, we prove that f(.F) and g(.F) have disjoint interiors when f ¥ g. 
But this is equivalent to F and g(F) having disjoint interiors for every 
2) = cea € I not equal to Id. Suppose first that m = 0. Then g(z) = z +k, 
for k £ 0, hence F and g(.F) have disjoint interiors. Now, suppose m # 0. Let z 
be in the interior of A. Hence |Re z| < 1/2 and |z| > 1. We then have 


|mz+n|? = m?|z2/? + 2(Rez)mn+ n? 


> m+n? — lmn| = (|m| — |n|)? + |mn| 


Since m # 0, the strict lower bound (|m| — |n|)? + Imn| is a positive integer. 
~k 
ere i 


if w=g(z) were in the interior of ¥, then the same argument yields 
Im g~!(w) < Imw. Hence Im z < Im g(z), and this contradiction shows that the 
interiors of F and g(F) are disjoint. Oo 


Consequently, |mz-+n|* > 1 and so Img(z) < Imz. Since g-!(w) = 


A remarkable feature of the animation in [15] is how new features are added 
near the bottom of the screen that clearly correspond to the bottom portions of the 
tessellation shown in Figure 6.13. 


6.3.1.5 Mappings to the Disk D 


We have emphasized the hyperbolic upper half-plane as a model for non- 


Euclidean geometry. Another model, which is equivalent to H, uses the unit disc 
D={zeEC: |z| < 1} as underlying set of points. Needham [20, pp. 317-318] 


shows that the map f(z) = an is a conformal map from 7 to D, and it induces 


a metric differential dsp on D given by 


ee 2 
dsy = ap ds 


(6.30) 


where ds*, = dx? + dy? is the Euclidean metric differential for D as a subset of 


the Euclidean plane E. With this metric for D, the map f : H — D is an isometry. 


Consequently, it maps geodesics in H to geodesics in D. The geodesics in D lie 
on either arcs of circles that intersect the unit circle at right angles, or diameters in 
D (which also intersect the unit circle at right angles). An excellent treatment of 
this disc model for non-Euclidean geometry can be found in Krantz [17]. More 
details about the geometry, including tessellations of D, are in Climenhaga and 
Kotek [5]. 

There is an abundance of artistic designs in D that have already been created. 
The designs by Escher, using curved polygonal tessellations of D are surely the 
most famous [8]. The mathematics of creating curved polygonal tessellations of D 
was worked out by Coxeter [6]. An entertaining app for creating your own 
designs, using Coxeter's tessellations of ID, can be found at the link in [3]. In 
Figure 6.14 we show two designs we created with this app. Design 1 used one 
of our Waratah flower rosettes as source image. It clearly retains the original 6- 
fold rotational symmetry of the original rosette. More importantly, from an 
artistic standpoint, it contains an ambiguity of figure-ground relations. If you 
view Design 1 from a far distance, the design features a star with six arms 
emanating from the central region of the design. However, when you move up 
close, this star fades into the background and six green/red flower-like regions 
closer to the circular boundary of the disc are more prominent. 


Design 1 Design 2 


FIGURE 6.14 


Two designs created with [3]. Design 2 includes tessellating curves, shown in white. Above Design 2 
are two zooms showing 3-fold symmetries in the hyperbolic disc D. 


Design 2 used one of our 6-fold symmetric wallpaper designs as source 
image. This latter design is particularly interesting in that it includes curved 
polygons (shown in white) that form a tessellation of D. The app [3] created this 
design by loading a clip of the source image into a central curved hexagon, shown 
in white at the center of Design 2. During the design process, it successively 
displays the iteration of isometries of this central, fundamental, hexagonal region 
to fill out the rest of the curved hexagons that are tessellating the disc. As with the 


tessellation of H we discussed above, there are centers of 2-fold and 3-fold 
symmetries in the completed design. It is easy to spot centers for 2-fold and 3- 
fold hyperbolic rotational symmetry near the top and bottom of the image. The 3- 
fold centers are located at intersections of the tessellating curves (see the two 
zooms on the right of Figure 6.14), just as they occur at such intersections for the 
tessellation of H. Notice that the original 6-fold symmetry of the source image is 
lost in Design 2, due to the clipping of only a part of the source image within 
the central curved hexagon. From an artistic standpoint, we like this symmetry 
breakage in Design 2. 


It is also an important fact that both 1/(1— |z|”) and ds? = dx? + dy? are 
invariant under rotations centered at 0. Since Jez)? = 1, we have the invariance 
of 1/(1 — |z|”) by rotation by 6, and the invariance of ds% under rotation holds 
because rotation by @ can be expressed as an orthogonal matrix. Since ds?, is the 
product of 1/(1—|z|?) and ds%, it is invariant under rotation about the origin. 
This rotational invariance is exhibited near the centers of both Design 1 and 
Design 2 in Figure 6.14, where these designs retain the rotational symmetry 
about the origin enjoyed by the rosette and wallpaper designs used as their source 
images. Similar 3-fold and 2-fold hyperbolic rotational symmetries, and 4-fold 
Euclidean rotational symmetry about the origin, are even more clearly evident in 
Escher's Circle Limit IIT woodcut [8] (if one ignores the different colors of the 
fishes). 


We have also produced images using conformal maps from H to D. For 
example, Stein [26, p. 211] shows that log z defines a conformal map from the 
half-disc {z=a+iy: |z|<1l,y>0}cD to the half-strip 
{w=u+iv:u<0,0<u< 7} CH. With this idea in mind, we created a 
design in D using a multiple of f(z) = log z + log(—z) and Blugold Fireworks 
as color map. It is shown in Figure 6.15. This design clearly shows 6-fold 
rotational symmetry about the origin in D and reflection symmetries through six 


diameter geodesics. We intend to continue exploring mappings from H to D, 
including ones that more fully exploit the isometric equivalence of H and D. 


FIGURE 6.15 
Design created using log z + log(—z). 


Conclusion 


We have shown a number of symmetric designs generated by the application of 
complex analysis to the non-Euclidean geometry of the hyperbolic upper-half 
plane and the hyperbolic disc. The history of the mathematics of hyperbolic 
geometry is a fascinating one. References for this history include Needham [20] 
and Greenberg [12]. Penrose [21, Chap. 2] has some fascinating insights. 
Mathematics related to hyperbolic geometry continues right up to the present day, 
see e.g., Adams [1]. 
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One of the most exciting results in the classical complex analysis is the Riemann mapping theorem which says 
that, except the whole complex plane, every simply connected plane domain is biholomorphically equivalent to the 
unit disc. Thus, the topological property “simply connected” is already sufficient to describe, up to 


biholomorphisms, a large class of plane domains. On the other hand, the Euclidean ball and the bidisc in C are 
topologically equivalent simply connected domains but they are not biholomorphic. This observation, which 
(under some additional assumptions) was made by H. Poincaré as early as at the end of 19th century, shows that 
even inside the class of bounded simply connected domains there is no single model (up to biholomorphisms) as it 
is in the plane case. To make the situation at that time more clear let us quote Carathéodory who wrote: “Die 


Ubertragung der Riemannschen Theorie auf Funktionen von mehreren Veranderlichen ist bisher aber nicht 
gelungen: Es liegen nur ganz vereinzelte und ziemlich diirftige Versuche vor, aus denen man hauptsdchlich das 
eine ersehen kann, daB ndmlich die Verhdltnisse hier ganz anders liegen, so dab man nicht einmal recht weib, wo 
man den Spaten anzusetzen hat, um die vermutete Goldader zu finden.” 

If two domains G, D C C” are biholomorphically equivalent, then they have the same bounded holomorphic 
functions, the same plurisubharmonic functions with a certain singularity, and the same amount of analytic discs 


(up to a biholomorphic map). Based on this observation it is important to associate with domains in C” tractable 
objects that are invariant under biholomorphic mappings. Provided that these objects are sufficiently concrete, one 
can hope to be able to decide, at least in principle, whether two given domains are biholomorphically distinct. We 
say that a system of functions (dp) ncen ss Geille EN? dp: Dx D—R,4, is a Schwarz-Pick system (or a 
holomorphically contractible system) of functions if it satisfies the following two conditions: 

a) dp equals the Poincaré distance of the unit disc; 


b) if DCC", GCcC™ are domains and F:D-—-G a_ holomorphic mapping, then 
dg(F(z), F(w)) < dp(z, w), z,w € D, ie. Foperates as an contraction. 

An object of this kind was introduced by C. Carathéodory in the thirties of the last century using the set of 
bounded holomorphic functions. In fact he defined a family of pseudodistances on domains via a “generalized” 
Schwarz Lemma. Thus, in particular, biholomorphic mappings operate as isometries. For such objects the name 
“invariant pseudodistances” has become popular. With the help of this pseudodistance he was able to give a simple 


proof of the fact that the ball and the bidisc in © are not holomorphically equivalent (without the additional 
assumption which was needed in Poincaré's argument)(see Section 7.1.3). Moreover, N. Kritikos exploiting the 
Carathéodory distance has shown that every biholomorphic mapping f from the domain 
D:= {z EC?: |z1| So |z2| = 1} onto itself has the origin as a fixed point, i.e., f(0) = 0, (see Section 7.1.9). 

It turns out that this family is the smallest possible Schwarz-Pick system of functions but also that it is suffers 
from some unpleasant properties which makes its discussion more difficult. For example: (1) the topology induced 
by the Carathéodory distance on a domain does not necessarily coincide with the standard topology of that domain, 
if the domain is unbounded; (2) the Carathéodory distance is, in general, not an inner distance which causes (at 
least at the moment) two different notions of “completeness”, even for a bounded domain (see Section 7.1.7). 

Discussing the family of plurisubharmonic function has led to the Green function which is not a 
pseudodistance but keeps the property that holomorphic mappings act as contractions with respect to the Green 
functions. Although the Green function plays an important role in modern pluripotential theory, we will only 
mention this family during the article but omit a detailed discussion (see Chapter 7.4). 


Besides using families of functions to associate (via an extremal problem) tractable objects with domains in C”, 
one can consider sets of analytic discs as new biholomorphic invariants. This idea is due to S. Kobayashi. While 
the family of Carathéodory pseudodistances is the smallest Schwarz-Pick system, the family of Kobayashi 
pseudodistances is the greatest one. Moreover, the difficulties for the Carathéodory distance mentioned above does 
not appear for the Kobayashi distance. 

For certain types of domains (convex domains or strongly linearly convex domains) it turned out that the 
Carathéodory and the Kobayashi pseudodistance coincide, a deep result which is due L. Lempert. Whether this 
equality would imply the convexity of the discussed domain was a long standing problem which only recently had 
a negative answer. At the end of Chapter 7.3 we will shortly present and discuss this example. 

Moreover, all these objects admit infinitesimal versions associating to any “tangent vector” a specific length 
contractible under holomorphic mappings. 

The main goal of this article is to introduce these concepts of “holomorphically contractible families of 


functions” for domains in C” and to emphasize how to apply them. 
Both authors thank S. Krantz for his kind invitation to write this survey. 


7.1 The Carathéodory Pseudodistance 


7.1.1 The hyperbolic geometry of the unit disc 


The unit planar disc D:= {A € C: |A| < 1} is the classical model domain for many problems of Complex 
Analysis. Recall that any simply connected domain whose boundary consists of at least two different points is 
biholomorphic to D (Riemann theorem, cf. e.g. [Con78]). For A’, € D let 


x = ava 
io MA" ; 


1, 1+m(X,2") 
MA"): & = <1 
p(n’, ") 2 °° 7 —m(N, 0") 


= tanh! (m()’, X”)) > m(X’, X”), where m(A’, A”) : & = | 


Remark 7.1.1. 


(a) One can check that p : D x D — R, is a distance. It is called the Poincaré (hyperbolic) distance. Moreover, 
m: D x D => (0,1) is also a distance (the Mdébius distance). 


(b) Recall that the group Aut(D) of holomorphic automorphisms of D consists of all mappings of the form 
>A ++ eh, (A) € D, where 6 € R, a € D, and h,(A) := ~=%. Observe that h,(a) = 0 and hj! = h_,. 
In particular, the group Aut(D) acts transitively on D. 


(c) The topology generated by p (resp. xsm) on D coincides with the standard Euclidean topology of D and the 
metric space (D, p) (resp. (D, m)) is complete. 


p(n") 


li _ li m()/,A") 1 
WAN NN 9d |A'-A"| _ WAN ANd |A'—A"| 


=p = yA), AE 


(d) 


Lemma 7.1.2 (Schwarz—Pick lemma, cf. [Rud74]) Let f € @(D,D) !. Then: 


(a) p(f(’), f(A") < pQ\', A"), 4, A" € D, ie. fis a contraction of the metric space (D, p). 


(b) The following statements are equivalent: 


(i) f ¢ Aut(D); 
Gi) p(f(d), F(A")) = p(X’, A"), 4, X” € D, ie. fis an isometry of the metric space (D, p); 
Gii) p(f(AQ)s F(AQ)) = P(A9; Ag) for some Aj, Ag € D with Ap A AG. 


ae) 


Remark 7.1.3. 


(a) Note that for any a,b € D, a ¥ b there exists a unique h = h,, € Aut(D) such that h(a) = 0 and 
h(b) € (0,1). 

(b) Fora,b€ D,a Fb, let agy(t) := h-(th(b)), t € [0,1], where h := h,y. Note that a,» : [0,1] > Disa 
curve, (0) = a, and a,»(1) = b. Observe that I,.4 := a4((0, 1]) lies on the unique circle C,» that passes 
through a and band is orthogonal to T := OD (if 0 € Iq4, then Cy, is a line through 0). 


(c) One can prove that the p-segment {c € D : p(a,c) + p(c, b) = p(a, b)} coincides with Ip. 
Indeed, since pis invariant under Aut(D), we may assume that a = 0, b € (0,1). We have to prove that 


{c € D: p(0,c) + p(c, b) = p(0, b)} = (0, d]. In other words, =A ee = ++ if and only if ¢ € [0,3]. 


Direct calculations show that the left hand side is equivalent to |c| < b and |c| = Rec. 


(d) The sets Ca» D play in the geometry of (D, p) the role of straight lines. Thus the fifth Euclidean axiom is 
not fulfilled in (D, p). Consequently, (D, p) is a model of a non-Euclidean geometry (the Poincaré disc model 
or Klein—Beltrami model). 


eax, Y) denotes the space of all holomorphic mappings f : X > Y; @(X) := 6(X,C). 


To see more geometric properties of the space (D, p) we apply the following procedure (cf. [Rin61]), which is 
known from metric analysis. Let d € {p,m}. We associate to any curve a : [0,1] — D (we always assume that 
curves are continuous) its d-length La(a) € [0, +00], 


N 
La(a) := sind So atta) :NEN, O0O=t9 <t) <-:-<ty= i} 
j=l 


Note that Lg(a) > d(a(0),a(1)). We say that a is d-rectifiable if La(a)<-+00o. We define 
d’':D x D-— (0, +0], 


d'(N,A"): = inf{La(a) : a: [0,1] > Dis acurve, X’ = a(0), X” = a(1)}, 
rr" e 


Clearly, d' > d. We say that dis inner if d' = d. 
Remark 7.1.4. In view of Remark 7.1.1(d) we get: 


(a) Lp = Lm and therefore p’ = m’, 
Indeed, by Remark 7.1.1(d), for any compact K C D and e« > 0 there exists a 6 > 0 such that 
0< p(’,rA”) — m()’, A") < eA’ — X"| for all A’, A" € K with |’ — X”| < 6. In particular, 
Lp(a) < Lm(a) + eL\ (a) for any curve a : [0,1] + K. Hence p‘ < m‘. 


(b) Acurve a: [0,1] — D is prectifiable if and only if a is mrectifiable if and only if a is rectifiable in the 
Euclidean sense. In particular, d' is a distance. 


dt. 


1 
Moreover, we associate to any piecewise @ !-curve a : [0,1] + D its y-length Ly(a) := / y(a(t))\a’(t) 
0 


Theorem 7.1.5. p(a,b) = p*(a, b) = Ly(aap), 4 6 € D, a ~b. In particular, p is inner, but m is not inner. 
Moreover, Lp(a) = Ly(a) for every piecewise @' -curve a : [0,1] > 


The following proposition shows that the non-Euclidean geometry of (D, p) fits perfectly with the holomorphic 
structure of D. 


Proposition 7.1.6. For any mapping f : D — D the following conditions are equivalent: 


Gi) fis a p-isometry, 
(i) either f € Aut(D) or f € Aut(D). 


7.1.2 The Carathéodory pseudodistance. The general Schwarz—Pick lemma 


We are going to extend the functions mand p to arbitrary domains G C C”. There are several ways to proceed. 
Here, we use the method based on bounded holomorphic functions (this way was the first also from the historical 
point of view; cf. [Car26], [Car27], [Car28]). For any domain G C C", n > 1, put 


moe(z',2"): =sup{m(f(z’), f(2")): fe O(G,D)}, 2,2" €G, 
ce(z’,2”"): =sup{p(f(z’), f(2")): f © 6(G,D)}, 2,2” €G. 


A standard Montel argument shows that Cg is finite. It is clear that cg = tanh ‘(mg) > mg and (by the 


Schwarz—Pick lemma) mp = m and cp = p. Observe that mc» = 0 (by the Liouville theorem). Obviously, we 
can always pass from mi to ¢g or conversely. In the sequel we will use both mz and Cg. 


We are not going to discuss the infinitesimal form yg of mg and Cg (like y in §7.1.1) in details. Nevertheless, in 


the sequel some proofs will require methods based on yg. Therefore, we define 


yo(z; X) := sup{y(f(z))|f'(2)X| :fEO(G,D)}, zEeG,Xec". 


Since m, p, and y are invariant under Aut(D), we get 


mg(z’,2”) =sup{|f(z”")|: f € 6(G,D), f(z’) =0}, 2,2" eG, 
cg(z’,z") =sup{p(0, f(z”)): f € G(G,D), f(z’) =0},  2’,z" EG, 
yo(3X)  =supt{|f'(z)X|: f € O(G,D), f(z) = 0}, 

Applying Montel's theorem, we find that for any z, z” € G there exists an f € O(G,D) such that f(z’) = 0, 


|f(2”)| = me¢(z’, 2”). Any such a function f will be called an extremal function for mg(z', z""). Since mand pare 
distances, the functions m, and ¢g are pseudodistances; mg is called the Mdbius pseudodistance forG, Cg is the 


Carathéodory pseudodistance forG. As a direct consequence of the definitions, we get 


Theorem 7.1.7. (General Schwarz—Pick lemma). For arbitrary domains GCC", DCC™ and for any 
holomorphic mapping F : G — D we have 


mp(F(z’), F(z”)) < me@(z',2"),  en(F(z’), F(z") < ee(z,2"), 2,2" €G. 
In particular, if F is biholomorphic, then the equalities hold. 
In other words, the systems (mg) ,, (€g) g are holomorphically contractible. 
Remark 7.1.8. 


(a) Observe that from the point of view of the general Schwarz—Pick lemma the Carathéodory pseudodistance is 
minimal in the following sense: 
If (dq) g is any system of functions dg : G x G —> R, where G runs on all domains in all C” 's, such that 
dp(F(z’), F(z")) < dge(z’, 2”), 2,2" € G, F € 6(G,D), anddp = m (resp. dp = p), thenmg < dg 
(resp. Cg < dg). 


(b) Note that in Theorem 7.1.7 we do not claim that mp(F(z9), F(z9)) = me(zp, 29) for some zo, 
zy € G, 2 # 2G, implies that Fis biholomorphic (cf. Lemma 7.1.2(b). This is not true even for D = GZC" 
and even under more restrictive assumptions on 2, 2) —take for instance D = G = D”; then using Theorem 
7.1.10(b) we easily conclude that |z1| = my»((0, 0), (z1, z2)) = mp2(F(0, 0), F'(z1, z2)) provided that 
|z1| > |z2|, where F'(z1, z2) := (21,0). 


(c) Using the Montel argument one can easily prove the following continuity property: 
If (Gk) py is a sequence of subdomains of G such that Gy, C Gpii, k € N, andG = UG, then 
mg, \ Mg, Cg, \ Cg, and Ye, \y Ya- 
(d) Similar to Remark 7.1.1(d), one can check that yg is a “strong” derivative of cg in the following sense: for any 


a € Gand X € C” with || X|| = 1? we have ye(a; X) = 


ee(z',2") 
I|z’—2" I 


. Note that for n = 1 the above 


li 
2! 2" a, 242" 
giz! 


\le!—2""| 


aX 


eo (22") 
|z!—2"| 7 


formula is equivalent to the following one: y¢(a; 1) = lim 
£2" 2! 2" a 


7.1.3 Carathéodory pseudodistance in balanced domains 


One of the natural questions of complex analysis of several variables is to find a sufficiently rich category of 
domains G Cc C” for which one may perform some effective calculations/estimates etc. The most natural one is 


the category of balanced domains. We say that a domain G C C” is balanced if for any z € G and \ € D the point 
Xz belongs to G. 


If Gis balanced, then denote by h the Minkowski function of G, h(z) := inf {t >0: tz EG }. Note that 
> Az) = |Alb(); 
- h is upper semicontinuous; 


° G=G, = {z€ C”: hz) < 1}. 


It is clear that in the case where n = 1 the only balanced domains are discs centered at 0, e.g. D = Dy with 


(A) = Il. 


Remark 7.1.9 Let G C C? be an arbitrary convex balanced domain. Then OG can be described in the following 
non-standard way which will be used in Section 7.1.9. 


2Here and in the sequel || X| := (|X1|?+...+ Xn)? denotes the Euclidean norm in C”. 


We associate to each point a = (a1, a2) € OG either the point (jar ay 1) € R, x C provided that a; 4 0, 


or the circle {0} x Taz provided that a; = 0. Observe that OG, and then also G, is completely determined by its 
image OG* under the above mapping. The set OG" is called the parametrization of OG. 


Using mainly the Hahn-Banach theorem one obtains the following result which helps to write down formulas 
for the Carathéodory pseudodistance for some domains. 


Theorem 7.1.10. Let G = Gy C C” be a balanced domain. 


(a) Forevery f € 6(G,D) with f(0) = 0, we have |f| < 6 in Gand |f'(0)z| < 6(z), z € C”. In particular, 
mg(0,-) <6 inG. 


(b) The following conditions are equivalent: 
G@) me¢(0,-) =) inG; 
Gi) ye(a;-) =h ine 
(iii) h is a seminorm, i.e. h(z + w) < h(z) + b(w); 


(iv) Gis convex. 


Note that for G = D part (a) is nothing else than the classical Schwarz lemma. 


Remark 7.1.11. For a balanced domain G C C” and a point a € G denote by A, an automorphism of G with 
h(a) = 0 (if it exists). Recall that: 


(a) if G = D”, then we may take hg(z1,..., Zn) := (a, (21), ---, Ba, (2n)); 
(b) if G =B,, and a ¥ 0, then we may take 


eae TTlal?*(\|al|22—(2,0)a)—|a?a+(z,0)a 


~~ lal? 1—-(2,a) : 
where (-, -) stands for the standard complex scalar product in C’. 


Corollary 7.1.12. If G = Gy is a balanced convex domain in C", a € G, and if h, € Aut(G) is such that 
h,(a) = 0, then m¢(a, z) = b(h,(z)), 2 € G. In particular, mp»(a,z) = max{m(a,,z;):j=1,...,n} and 
(=al?)(4=219) y? 


ma, (a, 2) = (1 “aye 


Remark 7.1.13. 


(a) Note that mp» is calculated via the maximum of the Mobius distances of the corresponding factors. In fact, it 
turns out that this kind of product property is true for arbitrary product-domains, i.e. the so called product 
property mp c((a, b), (z,w)) = max{mp(a, z), mg(b, w)} holds for arbitrary domains Dc C", Gc C™ 
(cf. [JP89]). 


(b) In the context of Corollary 7.1.12 one should point out that there are only a few classes of domains G for 
which ¢g may be effectively calculated. Besides the unit polydisc and ball, the formulas are known for 


example for complex ellipsoids, the symmetrized bidisc, and the Neile parabola (cf. [JP13], $§2.11, 7.1, 
16.6). 


Corollary 7.1.14. 


ilz'=2" || 
ar? 


(a) If B(a, 3r) C G, thenme(2z', 2") < mg (e127)(z', 2") = 
continuous. 


(b) If Gis a bounded domain, R := diam(G) (in the Euclidean sense), then 


"| 


z', 2" € B(a,r). In particular, ¢g is 


= le 
R 


mg(z', 2") > mgz.p)(z’, 2”) ,2',2" € G. Consequently, if G is biholomorphic to a bounded 
domain, then the topology induced by Cg is equivalent to the Euclidean topology of G, i.e. topeg = topG. 


Note that B, and D’ (n > 2) are topologically equivalent. Nevertheless, they are not biholomorphically 
equivalent as the next corollary shows. So the situation in higher dimensions differs strictly from the one 
dimensional case where the unit disc can be treated as a biholomorphically equivalent model for almost all simply 
connected domains. 


Corollary 7.1.15 (Poincaré theorem, cf. [Car26], [Rei21]). For n > 2 there is no biholomorphic mapping of D" 
onto B,. 


Proof. Suppose that F = (Fi,...,F,) :D” > B,, is biholomorphic. Since Aut(B,,) acts transitively on B,, we 
may assume that F(0)=0. Then, by Theorem 7.1.10(b) and Corollary 7.1.12, we have 


zn|?} = (mp»(0, z))? = (mz, (0, F(z))) = |Fi(z)|? +...+ | Fn(z) 2, 2€D". The right 


hand side is @~, the left hand side is even not differentiable; contradiction. 


prety 


max |21 2 


Remark 7.1.16. Poincaré has started the first steps into proving the corollary under the additional assumption that 
the biholomorphic mapping is regular up to the boundary. The above result has been proved by Reinhardt (see 
[Rei21]) without assuming this boundary condition. According to Reinhardt, Poincaré studied the question of the 


biholomorphically equivalence of three dimensional surfaces in C via establishing their groups of 


biholomorphism. Along this discussion he observed that there are domains in C? bounded by such three 
dimensional surfaces which cannot mapped biholomorphically onto the other one if the mapping is also assumed 
to be holomorphic up to the boundary. The proof above which as Carathéodory wrote does not use any calculation 
was, in fact, given by Carathéodory (see [Car26]). 


The above Poincaré theorem may be generalized in the following way. 
For a = (a1,...,a%) € N* let By := Ba, X ... X Bay. 


Theorem 7.1.17. (cf. [JPO8], Theorem 2.1.17). Let a = (a1,...,a%) € N*, B = (f1,..., Be) € N*. Then the 
following conditions are equivalent: 


(i) there exists a biholomorphism B, — Bg ; 


(ii) £=kanda = 8B up toa permutation. 


Moreover, every biholomorphic mapping F' : By — Bg 1s, up to a permutation of Bz,,...,Bg,, of the form 
F(z) = (Fi(a1),.--,Fe(2x)), 2 = (41,---, 2) € Ba, where F,, € Aut(Ba,), 3 =1,...,k. 


In the case where k= n> 2,a, =... =Qy) = 1, 2=1, 8, = 70 the result reduces to the Poincaré theorem 
(Corollary 7.1.15). 


Remark 7.1.18. The above theorem may be generalized to the case of Carathéodory isometries F : By — Bg (cf. 
Proposition 7.1.6). For more complicated domains the situation becomes unknown. 


7.1.4 Carathéodory hyperbolicity 


In general, the pseudodistance cg need not be a distance, e.g. Ccn = 0. Note that cg =0 if and only if 
F~(G) ~ C, where “#?™(G) ~ C” means that all bounded holomorphic functions on G are constant, i.e. G is 
a Liouville domain. On the other hand, ¢g is a distance if and only if the space #~(G) of all bounded 
holomorphic functions on G separates points in G. If cg is a distance, then we say that G is c-hyperbolic. By 
Corollary 7.1.14(b), if Gis biholomorphic to a bounded domain, then Gis e-hyperbolic. 

If G Cc C!, then Gis chyperbolic if and only if #°(G) 2 C. In other words, if a domain G C C? is not a 
Liouville domain, then it is c-hyperbolic. 

If n > 2, then there are domains such that cg # 0 but G is not chyperbolic. For example, take the convex 
balanced domain G := {z eC |z1 + z2| < 1}. 


7.1.5 The Carathéodory topology 


Recall (Corollary 7.1.14(b)) that if G is biholomorphic to a bounded domain, then topeg = topG. In C the 
situation is extremely simple, namely we have 


Proposition 7.1.19. IfG C C! is c-hyperbolic, then topeg = topG. 


Proof. Let Gda,;—>+>ae€G in the Euclidean sense. Observe that |f— f(a)| < ||f — f(a)||¢-me(a,-), 
f € ~(G). Consequently, f(a,;) > f(a) for any f € #~(G). Since #™~(G) > C, there exists an 
foc ~(G), fo # 0, with fo(a) =0. Write fo(z) = (z- ao)*9(2z), z€G, where g(ao) #0. Clearly, 
9 € H~(G). Since fo(as) > 0 and g(as) > g(a) 4 0, we have a, > ain topG. 

Unfortunately, for n > 3 there exist c- hyperbolic domains with topcg # topG. 


Theorem 7.1.20. (cf. [JPV91]) 
For any n > 3 there exists a c-hyperbolic domain G C C” such that topegZtopG. 


For n = 2 we do not know whether such a domain exists 3 


7.1.6 The Inner Carathéodory Pseudodistance 


Let d € {cg, mg}. Similarly as in §7.1.1, for a curve a: [0,1] + G we may define its d-length Lq(a). If 
a:[0,1} >G is a piecewise @1-curve, then we can also define its yg-length by the formula 


1 
Ly,(a) = i yc(a(t); a’ (t))dt. Moreover, we define 


d'(z',z") = inf{La(a) : a: [0,1] > G, aisacurve 


rectifiable in the Euclidean sense with a(0) =z’, a(1)=2"}, 2,2" eG. 


Observe that by Remark 7.1.4(b) in the case G = D the above formula gives p’ (resp. m1). 


Theorem 7.1.21. 


(a) Dmg = Lcg and, moreover, if a : (0, 1] — G is acurve rectifiable in the Euclidean sense, then 
Leg(a) < +00. 
(b) Ifa is piecewise @', then Le,(a) = Ly,(a). 


(c) Ifa: [0,1] > G isa curve rectifiable in the Euclidean sense, then for any € > 0 there exists a piecewise @' 
curve B : [0,1] — G such that B(0) = a(0), B(1) = a(1), and |Leg(a) — Leg(B)| < e. 


Corollary 7.1.22. 


(a) co = my. 
(b) ch is a pseudodistance and for any F € 6(G, D) we have c' 


HFC), #2) sch 2"). 2.2" €G, with 
equality for biholomorphic mappings. 


(C) e4,(z', 2”) = inf{Ly,(a) : a: [0,1] > G, a is a piecewise @' -curve joining z and z''}, 2', z" € G. 


We say that ch is the inner Carathéodory pseudodistance forG. Notice that the definition of ch is a little bit 
different than the one in §7.1.1. Here we take only those curves that are rectifiable in the Euclidean sense. We 
do not know whether in the definition of oe the Euclidean rectifiability of a may be omitted. It is known that 
this is possible in the case where G is y-hyperbolic, i.e. yo(a;X) > 0 for alla € G and X € C"\{0} (c.g. Gis 
biholomorphic to a bounded domain); cf. [Bar95] for a general discussion. 


In view of the equality p = p* one could conjecture that eg = c. Unfortunately, this not true as the following 
theorem shows. 


Theorem 7.1.23. (cf. [Vig83]). Let a, b € G, a 4 b. Suppose that there exists an f € @(G,D) which is extremal 
for mg(a, b) (f(a) = 0 ) and such that | f'(a)X| < y@(a; X) for all X € C"\{0}. Then cg(a,b) < ci(a, 6). 


Open problems are marked “ [?] re “te 


Example 7.1.24 (cf. [JP90]). Let P:= {z€ C:1/R < |z| < R} (R> 1). If g € G(P, D) is an extremal function 
for mp(1,—1), then the function f(z) = $(9(z) + g(1/z)), z€ P, is also extremal for mp(1,—1) and 
f'(1) = 0. Hence by Lemma 7.1.23 we get ep(1,—1) < c4(1,—1). 

Notice that even the following better result is true (cf. [JP91b]). 

Fora € (1/R, R) we have: c‘,(a, z) = ep(a, z) if and only if z € (1/R, R). 


7.1.7 Completeness—general discussion 


Recall that in the theory of Riemannian manifolds the concept of completeness for the distance induced by length 
has a successful history to clarify via this metric notion geometric properties of the manifold itself and vice versa 
(e.g. the result of Hopf and Rinow later in this section). So the study of completeness in the context of invariant 
distances may give some hope to get a better understanding of complex analysis problems via metric properties. 

It is well-known that the theory of holomorphic functions of several complex variables essentially differs from 
the theory of one variable. One of the main differences lies in the fact that each domain G C C is a domain of 
existence of a holomorphic function, i.e. there exists an f € O(G) such that there arenoa€GandO0<r<R 
with D(a,r) C G, D(a, R) 2 G, such that the function f|p(a,,) extends holomorphically to D(a, R) (D(a,r) 
stands for the disc centered at a and of radius t). If G = intG, then the above function fmay be even chosen in the 
space #™(G) (cf. e.g. [JPO0], §§1.7, 1.8). This is not longer true for domains G C C” with n > 1. For example, 
if Go C C” (n > 1) is a domain and K C Go a compact subset such that G := Go\K is a domain, then any 
function holomorphic in G extends holomorphically to Go (cf. [JP00], Theorem 2.6.6). For instance, one can take 


Go := B(R), K := B(r) with 0 <r < R (Hartogs’ Kugelsatz). These types of phenomena have led to the 
following three important notions. 
Let G C C” be a domain and let 0 4 F C O(G). We say that Gis: 


+ an F-domain of holomorphyif there are no a € G and 0 < r < R with B(a,r) C G, B(a, R) & G, such that 


for every function f € F there exists an f € O(B(a, R)) such that f = f on B(a,r) (cf. [JPOO], §1.7); if 
F = G(G), then we say that Gis a domain of holomorphy; if F¥ = #?~(G), then we say that Gis an #™- 
domain of holomorphy; 


« ¥-holomorphically convexif for every compact K C G the set Kg := {z EG:VyfegF: \F(2)| < fll} is 
also compact (cf. [JPOO], §1.10); if F=C (G), then we say that G is holomorphically convex; if 
F = 4~(G), then we say that Gis #~ -convex; 

« pseudoconvex if the function G3z ++ — log dist(z, OG) is plurisubharmonic. 


A geometric background of pseudoconvexity may be based on the following two facts (cf. [H6r94]): 
— If \varOmega¢R% is a convex domain, then the function \varOmega>z ++ — log dist(«, 0\varOmega) 
is a convex function, where the dist(«, 0\varOmega) is taken in the Euclidean sense. 


N 
O7u 


— A function u € @?(\varOmega,R) is convex iff ——_— 
k=l Ox jOX;, 


(x)X,;X, >0, « € \varOmega, 


X =(X,...,Xy) € R*. 

Summarizing, the property “pseudoconvex” can be seen as a kind of complex analogue of standard convexity. 

One can easily check that each domain G C C is a domain of holomorphy, holomorphically convex, and 
pseudoconvex. 

It is well known that in the case where FY = @ (G) the above three notions are equivalent (cf. [JP00], Theorem 
2.5.7). 


Observe (see Theorem 7.1.25) that in the case where ¥ = #°°(G) the first two notions are strictly connected 
with completeness of the space (G, cg). 


The implication (pseudoconvexity = > domain of holomorphy) is called the Levi problem and it was 
formulated by E.E. Levi in 1910 (cf. [Lev10]). The first positive answer has been given 30 years later in 1942 by 
K. Oka (cf. [Oka42] —the case n = 2) and in 1953 ({Oka53])—the general case). In the general case the Levi 
problem has been independently solved by H.-J. Bremermann (cf. [Bre54]) and F. Norguet (cf. [Nor54]). 


Proposition 7.1.25. Let G C C” be a c-hyperbolic domain. 

(a) Assume that any Cauchy sequence in the sense of Cg converges to a point in G with respect to the Euclidean 
topology. Then G is an #’™ -domain of holomorphy. 

(b) Assume that all balls (with finite radius) in the sense of ¢g are relatively compact in G in the Euclidean sense. 
Then Gis J? -convex and is an F¢™ -domain of holomorphy. 


In particular, in both cases G is pseudoconvex. 


Observe that the assumption in (b) is stronger than in (a). 

In the general situation let G be an arbitrary domain in C” equipped with a continuous distance dg, e.g. 
dg = cg when Gis chyperbolic. We point out that the d¢-topology may be different from topG. We distinguish 
four different notions of completeness. 


(C1) Gis weakly dg-complete if the metric space (G, dg) is complete. 


(C2) Gis dg-complete if any dg-Cauchy sequence in the sense of dg converges to a point in G with respect to 
topG. 


(C3) Gis d¢-finitely compact if all dg-balls (with finite radius) are relatively compact in G in the sense of topG. 

(C4) Gis weakly d,-finitely compact if all dg-balls (with finite radius) are relatively compact in Gin the sense of 
dg. 

Remark 7.1.26. 


(a) (C3) > topdg = topG. 
(b) (C3) = (C2) = (Cl). 


(c) In the case of the Carathéodory distance it is not known whether (C1) = (C3). A counterexample is known 
in the category of complex spaces (cf. [JPV93]). 


(d) The only well understood case is the planar case which was studied by M.A. Selby (cf. [Sel74]) and N. 
Sibony (cf. [Sib75]). 
Let G C C be c-hyperbolic. Then G is c-complete if and only if G is c-finitely compact. 


As it is known from differential geometry, a theorem of H. Hopf asserts the equivalence of Cauchy- 
completeness and finite compactness. This result was generalized by W. Rinow (cf. [Rin61]) and S. Cohn-Vossen 
(cf. [CV35]) to the situation we are interested in. 


Let dg be a continuous distance on G. Analogously as for the case dg = cg, we say that dgis innenf dg = di, 
, where 


di.(z',z) = inf {La,(a) : a: [0,1] > G, ais acurve 


rectifiable in the Euclidean sense with a(0) = z’, a(1) = 2"},2',2" EG. 


Remark 7.1.27. Let dg be a continuous inner distance on G. Then 


dg(z', 2”) = inf{La,(a) : a: [0,1] > G, ais continuous 
as a mapping into metric space (G, dg), a(0) = 2’, a(1) = 2"}, z’,2” €G. 


Consequently, dg is also inner the sense of Rinow (cf. [Rin61)]). 


Theorem 7.1.28. Let dg denote a continuous inner distance on G. Then topG = topdg and (C1) — (C2) => 
(C3) = (C4). 


Recall that unfortunately Cg is in general not inner. 


7.1.8 Carathéodory completeness 


Proposition 7.1.25(b) can be made more precise during the next result. 
Theorem 7.1.29. (cf. [Pfi84]) For a c-hyperbolic domainG C C” the following statements are equivalent: 


(ij) Gis c-finitely compact; 


(ii) for any zo € G and for any sequence (21) <x C G without accumulation points (w.r.ttopG ) in G there 


f(a) 


exists an f € O(G,D) with f(zo) = 0 andsup = 1, 
veN 


We say that a domain G C C” is C-convex if for any complex line L=a-+bC (a,b € C”, 640) with 
LOG #9 the intersection LM G is connected and simply connected (as a planar domain). Any convex domain is 


C-convex. It is known (cf. [Hér94], [APS04]) that if G is @?-smooth, then G is Cconvex iff 


(Nex) = Yo (x/K > 


= , z€ 0G, 
j,k=1 Oz;0Z% 


Oz; 
on an open neighborhood U of 0G satisfying UNG = {z € U: r(z) < 0}, U\G = {z eU:r(z)> of, and 
gradr(z) 4 0 for every z € OG). 


“9 
X €T£(AG) := {x éC"* Se  (2)X; - ob where ris a defining function for OG (i.e. ris a @?-function 
j=l 


Theorem 7.1.30. (cf. [KZ13]) For any bounded C-convex domain G C C” and any boundary point a € OG there 
exists an f € O(G,D) with im f(z) = 1. In particular, any bounded C-convex domain G C C” is c-finitely 
3z—a 


compact. 


It is known that if G is @?-smooth, then G is pseudoconvex iff (Yr)(z;X) > 0, z€ OG, X ¢ T£(AG), 
where ris a defining function (cf. [JPOO], §2.2). 

A bounded @?-smooth domain G C C” is called strongly pseudoconvex (cf. [JP00], Definition 2.2.4) if 
(Lr) (z; X) > 0, 2 € OG, X € TL(AG)\ {0}, where ris a defining function. 

A boundary point a of a bounded domain G C C” is called a peak-point with respect to a family F C C(G ) if 


there exists an f € ¥ such that |f(a)| =1 and |f(z)| <1 for ze G\{a}. It is known that if G is strongly 


pseudoconvex, then any boundary point is a peak-point with respect to @(G). If G is a bounded pseudoconvex 


domain in C with real analytic boundary, then any boundary point is a peak point with respect to @(G) 0 @(G) 
(cf. [BF78]). 


Remark 7.1.31. Using peak-functions we conclude that: 
(a) Any bounded strongly pseudoconvex domain in C” and any bounded pseudoconvex domain in C with real 
analytic boundary is c-finitely compact (cf. [BF78], [FM94], [FS81]). 


(b) Any bounded convex domain is c-finitely compact. 


Remark 7.1.32. 


(a) In the context of Proposition 7.1.25(b) it is not known whether the property “c-complete” implies that G 


is # ~©-convex 


(b) Observe that although the notions “#°°-domain of holomorphy” and “#°-convex” coincide for bounded 
plane domains (cf. [AS75]) they are not comparable in higher dimensions (cf. [Sib75]). 


(C) There exists a bounded pseudoconvex balanced domain Gin © with a continuous Minkowski function which 
is not c-complete although it is an 7 °-convex # °-domain of holomorphy (cf. [JP91a]). Other examples 
of domains sharing these properties were given by P.R. Ahern and R. Schneider (cf. [AS75]) and N. Sibony 
(cf. [Sib75]). 


(d) It is well known that any bounded pseudoconvex domain with smooth @ ~-boundary is #”~-convex and an 


IH ~-domain of holomorphy. [?] Despite much efforts no example has been constructed of a domain of this 


type not being c-finitely compact 


We say that a domain G C C” is Reinhardt (n-circled) if (e"z1,...,e""zn) € G for any (z1,..-,2n) CG 
and 6),...,0, € R. 

In the class of Reinhardt domains the c-completeness is completely understood and even expressed by a 
geometric condition. 


Theorem 7.1.33 ([Pf184]), [Fu94]), [Zwo00a]). Let G C C” be a pseudoconvex Reinhardt domain. Then the 

following conditions are equivalent: 

(i) Gis c-finitely compact; 

(ii) Gis c-complete; 

(ili) G is bounded and fulfills the following so called Fu-condition:if GM V; #9, thnGnV; #0, where 
Vie Cog =}. 

7.1.9 An application 


Shortly after Carathéodory had introduced his pseudodistance, Kritikos (see [Kri27b]) was able to prove the 
following result using this at that time new object. Looking at his proof one finds already a lot of ideas like 
complex geodesics and indicatrices which were also later used and made more explicit. Therefore we believe that 
the interested reader may enjoy to see some details of the proof. 


Theorem 7.1.34. LetD := {z € C?: |z1| + |z2| < 1} and let F € Aut(D), then F(0) = 0. 


Remark 7.1.35. (1) Observe that this result was received before (also by Kritikos) but under some additional 
assumption on the boundary behavior of F (see [Kri27a]). With this new tool at hand he was able to argue now 
only with the inner geometry of the domain D. 


(2) In the meantime there exists the same result for every domain D of the following form: 


p= {reo 3 


j=l 


5 


Pic i) where p; € (0,00)\{1}, j=1,...,n (see [JP13], Corollary 16.4.6). 


Moreover, using Lie theory the general case of bounded Reinhardt domains containing the origin is due to T. 
Sunada (see [Sun78]). 

(3) There are other examples of balanced domains which are even not Reinhardt such that any automorphism 
of it has the origin as a fix point, e.g. the minimal ball {z = (z1,..-,2n) €C”: |lal|? + jezi t+... +22| <1} 
(see [Kim91], [Zwo96]). 

(4) Even more obscure there exists a bounded convex domain in €* with C™~-smooth boundary such that 
Aut(D) = {idp} (see [Eic61]); this example solves a question posed by H. Behnke and P. Thullen in [BT70], 
chapter VI, §9.1. 


The proof of Theorem 7.1.34 needs several steps. The ones for which the Carathéodory distance is used will be 
explained in detail while the other steps mainly consisting of some calculations will be skipped. 


Lemma 7.1.36. Let GCC” be a domain and let p€ G(D,G) be a complex geodesic, i.e. 
ec(y(r’), e(A”)) = en(X’, A"), A’, A” € D, then ye((A); ¢'(A)(1 — |AI?)) = 1, A € 
Proof. We first assume that Ay = 0. We know that y(A) ¢ (0) for all A € D,. Take any sequence (A,)°, C Dy 
such that A, — 0. We know that OY oy 9, Passing to a subsequence we may assume that ds/|As| =e" 


Dal 
pl(As)—v(0) 
and 70,)=«y 7 a en 


1= lim = h 
5—+00 IAs s—+oo ||y(As)—9(0) 


= ya(¥(0); X)||e"(0)|I. 


In particular, p(0) 4 0. Now we have 


m((A5),0(0)) gs m((As),0(0)) | p(As)—9(0) | 
I Xs 


(As) (0) 
p(As)—9(0) rs 5 . eff g!(0 
x ) e€ 


= = ei ) 
wAs)-¥(0) ~~ TAs! || PAs Ile’(0) | 


=X. 


Finally, 1 = yg(9(0); X)||y’(0) || = ve(y(0); e%'(0)) = ve (v(0); ¢'(0)). 


Now let Ao € D be an arbitrary point. Then put (A) := y( an ), A € D. Thus (0) = y(Ao). Therefore, 
0 


1 = yo(¥(0); $'(0)) = va(e(Ao); e’(Ao)) (1 — |Aol*) 


which gives the claim in the lemma. 


Let G, g be as in the lemma and let ac€G. We define the c-indicatrix at a as 
I(G;a) := {X € C": ye(a; X) < 1}. Note that the indicatrix is a convex balanced domain in C” and if Ao € 
then y!(Xo)(1 — |Ao|?) € OL(G; g(Ao)). 

Recall (cf. Theorem 7.1.10(b)) that if D C C” is a convex balanced domain, then for every boundary point 
a € OD there is the following complex geodesic y_(A) := Aa, A € D. In particular, we have I(D;0) = D. 


’ 


Lemma 7.1.37. Let GC C” be a domain, a € G, and F € Aut(G), then F'(a)(I(G;a)) = 1(G; F(a)) ; in 
particular, F'(a) is a linear isomorphism from I(G; a) onto I(G; f(a)). 


Proof. Note that F’(a) defines a linear isomorphism from C” onto C”. Fix an X € I(G;a). Then 
1 > ye(a; X) = yc(F(a); F'(a)X). Thus F’(a)X € I(G; F(a)). Arguing now with F'~! instead of F, we see 
the inverse conclusion. 


To get a part of the indicatrix of D at a point a € D we will find some complex geodesics through a. From 
now on let I(a) := I(D;a). 


Lemma 7.1.38. [fr # s are real numbers of the closed unit interval (|—1, 1], then the map ,,; : D > C, 


Wrs(A) = fae. AED, 


is a biholomorphic mapping from D onto C\\{r, s] sending 0 to ~. Moreover, ,,; maps T onto the closed interval 
[rs]. 


h = 
Proof. Recall that the Joukowski transform D5, +> v1 is a conformal map from D onto C\[—1, 1] sending T 
onto [—1, 1]. Thus the map A +> r + (A(A) — 1) * does the job and this is exactly the map given by the formula 


2 
in the above lemma. 


Let &: C? > C? be given by P(z) := (21 + 22,21 — 22), 2 = (21, 22) € C?. Then y is a biholomorphic 
map from €@ onto C. Set D := &(D). Observe that D c D?. 


Our next aim is to find some complex geodesics in ®. Put Pr,s(A) = (A, AWr,s(A)), A € D. Note that G,,, is a 
holomorphic mapping. Then we have the following result. 


Lemma 7.1.39. If r # s are real points inside of |—1,1], then G,,,(D) C D C D? and G,., is a complex geodesic 
ind. 

Proof. (a) Note that u(z) := |z1| +|z2|, z € C2, is plurisubharmonic. Then h := u($~!°G,,,) is a subharmonic 
function on D which is not identically constant. Applying that h(e”) = 1 for all real ¢ and the maximum principle 


leads to the fact that h < 1 on D. Hence, %,,,(D) C D. 
(b) Fix now two points \’, A” € D. Then 


ep(’" r") 2 C5(Gr,5(X'), Pr,s(A")) = Cp? (Gr,s(A‘), @r,s(A")) 
= max{ep(A’, X"), ep(A'Wr,5(A’), A” Wr,s(A")) } = Epi’, A”). 


The last inequality follows using the Schwarz—Pick lemma. 


Applying that the Carathéodory distance is invariant under biholomorphic mappings we finally have: 


Corollary 7.1.40. [fr # s are two real numbers in |—1, 1], then 


AtAtrs(A)  AAdr,s(A 
Prg(A) = (AAO, AA) 


defines a complex geodesic in’). Or more explicit, 


Pr,s(A) = (T2.? =e + (1+ rts) 4 rs, st NP + +(1 res) 4 sr) 
With a similar argument as above we obtain a few more complex geodesics for D. 


Lemma 7.1.41. Leta € D. Then the mappings 


3A + (a1, (1—|a1|)A)€ D, DSA ((1— |a2|)A, a2) € D 


are complex geodesics inD. 


Proof. Because of the symmetry of D it suffices to study the first map; call it g. 
Put V:D >C?, Wz) :=(z, ics ). Note that y is an injective holomorphic mapping with 
U(D) =: D C D*. Then G(A) := (ai, 


(1-]a1|)A 
1-a; 


) maps D to ® and also to D2. Therefore, if X’, A” € D, then 


ep(N',r")_ > ¢5(B(N’), P(A") = ewe (G'), BO") 
= max{ep(a1, a1), ¢p(X', A”) } = ep(A’, A”). 


Using Y~! finishes the proof. 


Now fix a point a 4 0 in D. We are going to describe certain complex geodesics through a. Because of the 
symmetry of © we assume that the as are non negative real values with 0 < aj + a2 <1. 


Lemma 7.1.42. Leta € D as before. Whenever s € [—1,1], s 4 “—“, then 


a,+ay ’ 


._ 4(a1—a2)+s(1—a1—ay)” 
r(s) = (1+a)+a2)" [-1,], 


r(s) al 5, Pr(s),s(@1 + ay) =a, 


ie. there is a one parameter family (p,s),, of complex geodesics in‘ through the point a. 


Proof. A simple calculation gives immediately the result. 


Hence we obtain the following boundary points of the indicatrix I(a). 
Corollary 7.1.43. Let a be as above and Xo := a1 + ao. Then all the following points 
(a) P1(s),s(0)(1 — }A0?) € OI(a), s as above, 


() (0,(1—a)(1— —*,)), 


(1-a1)? 


(©) ((1 — a2)(1 — —*,), 0) 


(1-a2)? 


belong to OI(a). 


Moreover, a simple calculation gives the following explicit form of the points in (a): 


are (1 + 8a, — a2 + s(1 — a; — ag), 1 — ay + 3a2 — 8(1 — a; — a2)), 5 as above. 
Then the following set M, described in (a), (b), and (c) below, belong to the parametrization of OI(a), where 
0<a,t+a2<1l,a; € Ry: 
(a) 4-2 (1 + 8a; — a2 + s(1 — a1 — ag), 1 — a1 + 3a2 — 8(1 — a1 — az)), 5 as above; 


(>) 0, 4(1 — a)(1- —2,)); 


(1-a3)? 


© \(Geapt -— 4) 0). 


(1—a2)* 


Analyzing the set in (a) gives that it consists of an interval on the line ? given through the 


a=—y+1—(a; +42) 
(7.1) 
with the following endpoints 
((1 + a1 — ag)(1 — a1 — a2), 2a2(1 — a — a2)), 
(2a1(1 — aj — ag), (1 — a1 + a2) (1 — a — a)), 


which lie in the open first quadrant except the case when a; = 0 (resp. ag = 0). 

Moreover, the point in (c) lies on the y-axis between the projections of the first endpoint and the point, where ¢ 
cuts the y-axes. On the other hand, the point given in (b) with positive second coordinate lies on the positive a-axes 
between the projection of the second endpoint given above and the point, where ¢ cuts the a-axes. The remaining 


point in (b) is just opposite. Therefore, it needs at least three lines each of which cuts the set M to cover M, and no 
pair of these lines are symmetric with respect to 0. 
The remaining step consist of the following claim which immediately implies Theorem 7.1.34. 


Lemma 7.1.44. There is no linear isomorphism A : I(0) + I(a), ifa€ D,a 40. 


Proof. Assume the contrary. Then there exists a linear isomorphism of C onto itself given by the inverse of A, 
that sends OI(a) onto OJ(0). Let us assume that 


W(z) = (pzy + qz2,uz1 + vz2), 2€ C’, 


where p=pit+ip,g=qt+ig,u=wutiu,v=wutineC. If (y,a+%8)€OAl(a)*, then 
W(y,a +78) € OI(0). More concrete, 

W(y,a+iB) =(A+iB,C+iD), where 

A:= ryt+una-@s, B:=pryt+qub+ qa, 

Cs ryy +uya— v2.8, D:=usy+v2a4+ 018. 


Or, if (A, B) ¥ (0,0), then the point 


(V/A? + B?, (C+ iD)(—4 iam) 


VAR? 24 BP 
is the corresponding point in 0J(0)*. If A = B = 0, then the corresponding point is given by (0, C + iD)T. 
12 
Using that this new point belongs to J(0) it follows that C? + D? = (1 — VA? + B®)’, resp. C? + D? = 1. 
Applying now this equation to those points in OI(a) with 6 = 0 leads to the following equality 
A((pry + aia)” + (poy + q20)") 
= (1+ (pry + qa)” + (poy + G2a)”) — (wry + v1)” — (uay + v20)”)?. 
(7.2) 
Recall that a non trivial interval on some line, given by the equation a = —y+c, belongs to part 8 = 0 of 


OI(a)*. Therefore the former equation after substituting a by c — ¥ is true for all y € R, ie. its left side is the 
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square of a polynomial. Hence one obtains that 0 = det @ ae ae “) = det & ) . Therefore, 
(pry +q1a)” + (poy + q2a)” = (Py + Qa)’ for all o and y. Then equation (7.2) is of the form 


(ury + via)? + (uy + va)? = (1+ (Py+ Qa))’. 


A similar argument as before leads to 


(ury + 11a)” + (uy + wa)” = (Ry + Qa)’, 7,0 ER. 


Finally, (7.2) reads as follows: (Ry+Qa)?=(14+(Py+Qa))’. So one obtains that 
OI(a)* NM (Ry x (R+70)) consists of at most four lines pairwise symmetric to the origin. But this was already 
impossible for that part of OI(a)* 7 (R, x (R+70)) which was discovered in (7.1); a contradiction. 


As a final consequence there is the following complete description of Aut(D). 
Theorem 7.1.45. [fD := {z ECc?: |z1| + |z2| < 1}, then Aut(D) = { Pa,Bs Py, : a, B,y,6 € R}, where 


Pa,e(z) = (e*%z1, e220), ,5(z) = (e"z2,e% 21), z= (21,22) € D. 


Note that the behavior of Aut(D) is extremely opposite to the one of the bidisc or the ball. 


To present the proof of Theorem 7.1.45 the following lemma is needed which, in fact, is a special case of a 
theorem of H. Cartan (cf. [Car30)]). 


Lemma 7.1.46. If F = (fi, fo) : 2 — D be holomorphic with F(0,0) = (0,0), 5=-(0, 0) =1, 4 (0, 0) =0, 
Sh (0,0) = 0, $2(0,0) = 1, then F = idg. 


Proof of Theorem 7.1.45. Let F € Aut(®). Then the mapping F; defined as F(z) := F-'(2F(¢z)), CeT 


and z € 9, is an automorphism of which satisfies the assumptions of the former lemma. Hence, F = idg. Or 
in other words one has that F(¢z) = ¢F(z) for all ¢ € T and z € D. Using the expansion of F into homogeneous 
polynomials leads to the fact that Fis a linear isomorphism of ©. Taking the form of into account the claim in 
the theorem is now an easy consequence. 


Finally, let us quote another result due to H. Cartan (see [Car36]) whose proof is also based on the 
Carathéodory pseudometric. To quote H. Cartan: “Pour la demonstration nous utiliserons la métrique de 
Carathéodory”. 


Theorem 7.1.47. Let G; C C™, j = 1,2, be bounded domains. Then for every ® in the connected component of 
Aut(G) that contains the identity, there exist 6; € Aut(G,;), 7 =1,2, such that ®(z,w) = (f(z), $2(w)), 
z€Gy,we Go. 


Looking back one can already here say that certain problems in several complex variables could be 
successfully handled using the language of metric geometry. 


ee 
7.2 The Kobayashi pseudodistance and the Kobayashi-Royden pseudometric 


In the previous chapter, we discussed the Carathéodory pseudodistance for domains G C C” using an extremal 
problem for @(G,D). About thirty years later, S. Kobayashi introduced another pseudodistance based on the 
family @(D, G), the set of so-called analytic discs in G. We already learnt that the Carathéodory pseudodistances 
form the smallest family of pseudodistances being a Schwarz-Pick system. It will turn out that the new family is 
the largest possible one. 


7.2.1 The Lempert function 


First we introduce a family (€g)¢ of functions £g : G x G — R, from which the new pseudodistance will be 
derived as the largest pseudodistance below fg. 
Before we are able to present the formal definition, we have to make the following observation. 


Remark 7.2.1. Let G be a domain in @” and z’, 2” € G. Then there exists a curve a : [0,1] + G connecting the 
points z, z. Using the Weierstrass approximation theorem we find a polynomial map P: [0,1] > G with 
P(0) = 2’ and P(1) = z”. Then it is easy to choose a bounded simply connected domain D Cc C, [0,1] C D, such 
that P(D) Cc G. By the Riemann mapping theorem we conclude that z,z lie onan analytic disc y : D > G with 
(0) = z' and y(c) = z” for some 0 < o < 1. Summarizing we see that any two points in a domain G C C” lie 
on an analytic disc in G. 


Definition 7.2.2. Let Gbe a domain in "and z',z” € G. Put 


lg(z',z") = inf {pQ’,A") NA" € D: Juco 2 GO’) = 2, A") = 2"} 
= inf {p(0, MM) 2M" ED: ALeee) ? 9(0) = 2, e(A") = ZY, 
La := tanhég. 


£q is called to be the Lempert function forG. 


Intuitively, the larger a disc in G containing the points z’, z” the smaller the value of &¢(z’, 2”). For example, 
Len(z', 2”) =0 for all pairs of points 2’, 2” € C” (pr(A) := 2’ + RA(z" — 2’) is an analytic disc in C” with 
yr(0) = 2’ and yR(1/R) = 2"). 

Moreover observe that: 

(a) €g: Gx G— R, is asymmetric function; 


(b) (€¢)g is a contractible family of functions with respect to holomorphic mappings, i.e. if F € G(G, D), then 
Lp(F(z’), F(z") < le(2, 2"), 2,2" €G; 

(c) in particular, we have £p(F(z’), F(z”)) = £¢(z', 2”) whenever F' : G + D is a biholomorphic map; 

(d) £p = p (use the Schwarz—Pick lemma); 

(e) eg < fe; 

(f) if there is a family (dg), of functions dg : G x G — R, contractible under holomorphic mappings with 


dp =p, then dg < £g. In other words, the system (£¢) q is the largest Schwarz-Pick system of functions. 


The following example shows that, in general, the Lempert function does not fulfill the triangle inequality; thus 
it is not a pseudodistance. 


Example 7.2.3 (cf. [PS89]) Let G:= {z€ C?: |z1z2| <1, |z2| < 10} and let a := (1,0), b:= (0,1) be two 
points in G. Then using the above properties one easily sees that the following equality is true: 


Lo(a, 0) + £a(b, 0) = £c(0, 1) + £p,10) (0, 1) = £10p(0, 1) = £p(0, 1/10) = p(0, 1/10). 


On the other hand there is the following lower estimate of £¢(a, b): let y = (~1, 2) € G(D, G) be an analytic 
disc with y(0) = a and ¢(s) = 6, where s € (0,1). Since ~i(s) = 0 and ¢2(0) = 0, the functions g; may be 
written as y1(A) = (s — A)G1(A) and y2(A) = AG2(A), A € D, where G; € O(D, C). Note that G1(0) = 1/s and 
G2(s) = 1/s. 

Take an arbitrary r€(s,1). Using that |yiy2]}<1 on G, the maximum principle leads to 


1 
1-s° 


Now observe that the function $2 — G2(0) € @(D,20D) and vanishes at zero. Hence Schwarz's lemma 
implies that |@2(A) — G2(0)| < 20|A| for all A€ D. Or |G2(0)| > |G2(s)| — 20s = 1/s — 20s. Moreover, 
1219) = |P1(0)%2 (0)| < si. Combining the last two inequalities leads to 1 — 20s? < = or after some simple 
calculations to f¢(a,b) > p(0,1/5) > p(0,1/10) = f¢(a,0) + £¢(0,b), which contradicts the triangle 
inequality. 


ALA)B)| < ae as long as |A| < r. Or when rtends to 1, then |F1Go| < 


Remark 7.2.4. Already here we draw the attention of the reader to the fact that convex domains will play a non- 
expected important role during the discussion of invariant functions. For example, for a convex domain G C C” 
the triangle inequality for the Lempert function 2g is true. 

Indeed, fix a convex domain G EC and three pairwise different points z,w,s € G. Then, if ¢ > 0 is given, 


there exist analytic discs y,w¢O(D,G) and numbers 7,0 € (0,1), 7<o, such that y(0) =z, 


A-0)(A-1/o 
y(t) =w = V(t), Y(o) = s, and £g(z,w) + € > p(0,7), £e(w, s) + € > p(T, a). Define x(A) := ee 


A € C\{r, 1/7}. Obviously, y is holomorphic on U\{7} for an open neighborhood U of D, having a simple pole 
at t. Moreover, observe that x(T) Cc [0,1). Finally, define the “convex” combination h := (y — #)x + W. Since 


y(t) = (7), it turns out that A is holomorphic on D satisfying h(T) C G and h(0) = z. Applying some 
maximum principle, it follows that h € @(D, G) with h(0) = z, h(o) = s. Therefore, 


La(z,s) < p(0,c) = p(0,7) + p(t, 0) < lg(z,w) + La(w, s) + 2, 


which proves, since ¢ was arbitrarily chosen, the triangle inequality. 
Later on (cf. Chapter 7.3) we will learn that, due to a very deep result of L. Lempert [Lem81], [Lem82], 
[Lem84] and H.L. Royden & P.M. Wong [RW83] even more is true, namely 2g = cg for such domains. 


Moreover, the Lempert function is in general not continuous. 


Proposition 7.2.5. Let G := {z © C”:h(z) < 1} denote a balanced pseudoconvex domain with Minkowski 
function b, i.e. h : C" — [0,+00) is a plurisubharmonic function with h(Az) = |A\h(z) (A € C, z € C” ). Then 
£e(0, z) = p(0, b(z)), z € G. In particular, £% (0, z) = |lz\|, 2 € Bn, and €p,(0, z) = max{|z;|: 7 =1,...,n}, 
ze D”". 


The proof is based on the maximum principle for subharmonic functions. 

Since there are many bounded balanced pseudoconvex domains whose Minkowski functions are not 
continuous, Proposition 7.2.5 shows that the Lempert function is in general not continuous, even not as a function 
of one variable. 

But £g is always upper semicontinuous. 


Proposition 7.2.6. The Lempert function £g is upper semicontinuous. 


Recall that £g¢ may be defined via analytic discs y € @(D,G), i.e. discs with a compact image y(D) in G. 
Hence a small deformation of such a g easily leads to the former result. 
Although those defects described above one has to emphasize that, in general, it is easier to handle £g than cg. 


To conclude this paragraph some other properties of the Lempert function will be summarized. In order to be able 
to calculate the Lempert function the following results turn out to be helpful. 


Proposition 7.2.7. Suppose that two domains G C C” and D C C™ are given. If z',z" € G and w',w" € D, then 
the following formula 


Lexp((z',w'), (2”,w")) = max{lg(z', 2"), £p(w', w”)} 
is true. 


It is said that the family of Lempert functions satisfies the product property. It should be emphasized, that the 
proof of Proposition 7.2.7 is much simpler than the one for the product property of the Carathéodory 
pseudodistances. 

Finally, we mention how the Lempert family behaves under an increasing union of domains. 


Proposition 7.2.8. Let G,G; C G, j © N, be domains with the property thatG;_/ G. Then £g, \, ta. 


7.2.2 Tautness 


Although the Lempert function need not to be continuous, there is a sufficiently rich family of pseudoconvex 
domains whose Lempert functions are continuous. 


Definition 7.2.9. Let \varOmega Cc C” be a domain. Then Q is called tautif the space @(D, \varOmega) is 
normal, i.e. whenever we start with a sequence (f;);°, C O(D, \varOmega), then there exists a subsequence 


co 
Vv 


(fj) p24 with f;, > f © @(D, \varOmega) locally uniformly in D or there exists a subsequence (f;,)”°, which 


diverges uniformly on compact sets, i.e. for any two compact sets K C D, ZL c \varOmega there is an index vo 
such that f;,(K) NL = Qifv > v9. 


Before we continue our investigations, we make a small digression on tautness. We only collect the most 
important results on this subject omitting their proofs. 


Remark 7.2.10. 


(a) Any taut domain is pseudoconvex. 


(b) Any bounded hyperconvex domain G C C”, i.e. any bounded domain G for which there exists a negative 
plurisubharmonic exhaustion function y : G —> (—oo, 0), is a taut domain. In particular, any bounded convex 
domain is a taut one. Note that a hyperconvex domain is pseudoconvex. 


(c) Any bounded pseudoconvex domain with % !-boundary is taut. We point out that these domains are also 
hyperconvex (cf. [KR81]) so that (b) yields tautness, too. 


(d) In the case of a balanced pseudoconvex domain G := {z € C” : h(z) < 1} with Minkowski function 6, there 
is even a complete characterization of tautness (cf. [Bar83]), namely: 
G is taut if and only if h is continuous with h > C\| || for a suitable C > 0. 


(e) Let 1: D—- G bea holomorphic covering, i.e. for any point a € G there exists an open neighborhood 
U = U(a) C G such that IJ~!(U) is the union of pairwise disjoint open sets V, C D such that I]|y, is a 
biholomorphic mapping from V, onto U. Then Dis taut if and only if Gis taut (cf. [Bar71]). 


(f) The above property and the uniformization theorem in classic complex analysis immediately imply the 
following result. 
A domain G C C is taut if and only if ##(C\G) > 2 if and only if there exists a holomorphic covering 
T:D—>G. 


Now, we come back to our discussion on the Lempert function. Namely, we want to study this function on taut 
domains. Our first result is the following one. 


Proposition 7.2.11. Let z, z be two points of a taut domain G C C”. Then there exist a holomorphic map 
y € O(D, G) and a number o € (0,1) with g(0) = 2’, p(o) = 2”, and £¢(z', 2") = p(0,c). Such a @ is called to 
be an extremal disc in G passing through the points z', z" . 


Proof. By definition we find a sequence (95) ja Cc G(D, G) satisfying 


;(0) = 2’, pj(0;) = 2" (0< 0; <1), and p(0,0;) \ le(z’, 2”). 


Since G is a taut domain and (0) =z’, we can choose subsequences ((;,)~, C (%j)j2, and 
(03,024 C (oj)2, with y;, +e € O(D,G) locally uniformly in D and oj, > € [0,1). From this we 
conclude that y(o) = 2”, y(0) = 2’, and p(0,c) = £¢(z’, 2”). Hence, we have proved that there always exist 
extremal discs through two given points. 


Remark 7.2.12. (a) Note that the claim of Proposition 7.2.11 is not longer true if Gis not taut. For example, take 
Go := B2\{(1/2,0)} and 2! := (0,0), 2” := (1/4, 0). Using the analytic discs D>A > (RA, s(R)A(A — ae)), 
R <1, s(R) <1, one can easily deduce that €¢,(z’, z”) < p(0, 1/4). 

Now suppose that there exists p=(v1,¢2)€ 6(D,Go), y(0)=2', yv(o)=z" such _ that 
£g(z', 2”) = p(0,c). Thus o < 1/4. On the other hand, the Schwarz lemma implies that 1/4 < o. Hence 
yi(1/4) = 1/4, ie. yi(A) = A. Since y € G(D, Bz), it turns out that y2 = 0. In particular, y(1/2) = (1/2, 0) 
which contradicts the definition of G. 

(b) Let D C C” be an arbitrary domain. Observe that if ¢ € @(D, D) is an extremal disc through two different 
points z', 2” € D with y(0) = z' and y(o) = 2", 1.0 < o = €5,(z’, 2”), then y(D) cannot be relatively compact 
in D. Otherwise one could find a number @ € (0, 1) and a new analytic disc G(A) := y(A) + 4 ((c) — y(6a)), 
A € D, in D. Then G(0) = z' and G(90) = z"; a contradiction. 


Moreover, we have the following result on the continuity on &g. 


Proposition 7.2.13. If G is a taut domain in C”, then the Lempert function £g is continuous onG x G. 


7.2.3 The Kobayashi pseudodistance 


To overcome the difficulty connected with the triangle inequality we modify the function £g in such a way that the 
new function becomes a pseudodistance, the largest one below of &g. 


Definition 7.2.14. Let G Cc C” be a domain and 2’, z” € G. Put 


ke(z2’, 2") : 
N 
= inf S 5 fe(zj-1, 23) :NEN, 2 =2', 21,.--,2n-1 € G, zn = 2" 5, 
=1 


j= 


kG := tanhkg. The function Kk; is called the Kobayashi pseudodistance for G. 
k, was introduced in 1967 by S. Kobayashi (see [Kob67]). 


Remark 7.2.15. Notice that the following properties hold for the system (kg) ¢: 


(a) Kgis a pseudodistance on G; 
(b) even more, kz is the largest minorant of &¢ that satisfies the triangle inequality; 


(c) if F € O(G, D), then kp(F(z’), F(z”)) < ke(z’, 2”), ie. the system (kq) ¢ is contractible with respect to 
holomorphic mappings; 


(d) kp = £p = p. Even more, we have: 


(e) if (de)g is a system of pseudodistances dg : G x G — R, with the properties stated in (c) and (d), then 
dg < ke; 


(f) in particular, cg < ke. 


To be able to continue the discussion on the Lempert function and the Kobayashi pseudodistance we need at 
least a few examples for which these objects can be calculated. 


Example 7.2.16. Let be a seminorm on C”. Denote by G := {z € C”: h(z) < 1} the associated open unit h- 
ball. Then kg(0, z) = £¢(0, z) = p(0, h(z)), z € G. In fact, by Corollary 7.1.12 and Proposition 7.2.5 we have 
p(0, 5(z)) = ea(0, z) < ka(0, z) < (0, z) < p(0, h(2)). 


In particular, we mention the following special cases. 


Example 7.2.17. (a) kp» (0, z) = £p»(0, z) = max{p(0, |z;|): 1 <j <n}, (b) 
kg, (0, z) = £8,(0, 2) = p(0, |l2I\). 


As a consequence of this example we obtain (cf. Corollary 7.1.14) 
Proposition 7.2.18. The functionkg : G x G — [0, +00) is continuous. 


7.2.4 General properties of k 


We already know that the || ||-topology of a domain G C C” is stronger than the k;-topology on G. We remember 
that in the case of the Carathéodory distance the || ||-topology can be different from the e¢-topology. To discuss the 
analogous question for the Kobayashi distance we need the following observation; see [Kob73]. 


Proposition 7.2.19. The Kobayashi pseudodistance is inner, i.e. if z, z" € G, then 


ke(z’, 2") = inf{Ly,(a): a: [0,1] —>G 


is continuous and || ||-rectifiable with a(0) = z’, a(1) = z’}, where 


N 
Lyg(@) = nif Sokola( a) a(6) :NEN, 0O=t) <t1<...<ty= 7 
j=l 


denotes the kg-length of a. 


Recall that, in general, the formula analogous to that of Proposition 7.2.19 fails to hold for the Carathéodory 
pseudodistance. 
It is obvious that a necessary condition for the || ||-topology and the k-topology to be equal is that kg is a 


distance. We say that adomain G C C” is k-hyperbolic if its Kobayashi pseudodistance is a distance. 


Remark 7.2.20. By the well-known inequality cg <kg it is clear that any chyperbolic domain is also & 
hyperbolic. Therefore, any bounded domain Gis Khyperbolic. 


So far we have shown that the Kobayashi pseudodistance is continuous and inner. These conditions suffice to 
prove the following comparison property of the topologies (cf. [Bar72]). Recall the situation for the Carathéodory 
pseudodistance which was more complicated. 


Proposition 7.2.21. If G is a k-hyperbolic domain in C", then its || || -topology is equal to the kg-topology. 


Proof. Since Kg is continuous, every k-ball is a || ||-open set. Suppose now that there is a norm-ball B(a,r) CC G 
which is not K,-open. Then there exists a point b € B(a,r) containing no kball. Therefore one may find points 
z; € G\B(a,r), 7 € N, with s; := kg(b, z;) > 0. 

Using the Khyperbolicity of Gand the continuity it follows that kg(b, z) > s > 0, z € OB(a,r), for some s. 
Now one applies the innerness. Thus for every j there is a curve y; in G connecting 6 with z; such that 
Lg (73) < 28;. Take a point a; € OB(a,r) lying on the th curve. Then 2s; > kg¢(b,a;) +kg(a;,z;) > 8; a 
contradiction. 

It should be mentioned that for a bounded domain G this result follows directly from Exercise 7.2.17. 


Applying that the Kobayashi pseudodistance Kg is the largest pseudodistance below of £g, Propositions 
7.2.8 and 7.2.7 immediately lead to the two following properties. 


Remark 7.2.22. 


(a) Let G = U G,, where (G_)°°, is an increasing sequence of domains in C”. Then for z, z" © Gwe have 
v=1 
Ea(z’,2") = lim Lg, (2, 2"). 
Vv—-oo 
(b) The Kobayashi pseudometric satisfies the product property. 
In calculations of the Kobayashi pseudodistance holomorphic coverings often play an important role because 


of the following result of S. Kobayashi (cf. [Kob05]) whose proof is based on the lifting property for holomorphic 
coverings. 


Theorem 7.2.23. Let II: G— G be a holomorphic covering. Then for x, y € G and % € G, II(#) = 2, the 
Lempert function and the Kobayashi pseudodistance for G satisfy the following formulas: 


le(a,y) =int{lq(z,9):9¢ G, Ug) = yh, 
ko(e,y) =int{ke(%,9):9¢ G, 1@) =}. 
Remark 7.2.24. 


(a) From the previous theorem we conclude that if all k a balls with finite radii are relatively compact subsets of 
G, then for x, y € G, and % € G, IT(%) = «, there exists a point § € G with I7(g) = y and 
kg(z, y) = kg@(Z, 9) (the same statement is also true for the Lempert functions provided that G is taut). 

(b) Notice that in general Theorem 7.2.23 is not true in the following strong sense, namely, that there exists a 


point g € G with kq(%,9) = ke(z, y) (this was a question posed by Kobayashi; the counterexample is due 
to W. Zwonek). 


(C) Since any domain G C C has D or Cas its universal covering, Theorem 7.2.23 also implies that £¢ satisfies 
the triangle inequality; hence kg = ég. 

(d) We also mention that for G := C\{0, 1} we have cg = 0 whereas kg(z’, 2”) > Oif 2’ 4 z”;i1e. Gis k 
hyperbolic. The latter fact follows because of Theorem 7.2.23 and the well-known result that D is the 
universal covering of G. 


(e) Moreover, the covering result may be used to obtain a formula of the Kobayashi distance for the punctured 
disc D,, namely: if a € (0,1) and z € D, with z = |z|, 0 € (—7, 7], then 


= 1 8 +(log |2|—log a)” 
kp, (a, z) = tanh (Fs (ioe el Hog a)? ) 


So far we know only few examples of domains G for which cg # kg. For plane domains we have the 
following complete characterization of such domains. 


Proposition 7.2.25. 


(a) Let Gbeac-h yperbolic domain in C and let us suppose that there is at least one pair of different points z’, 
2" €G withkg(z’, 2”) = eg(z’, 2”). Then G is biholomorphically equivalent to D and sokg = cg. 


(b) Ifa plane domain G is not c-hyperbolic, then cg = 0 and eitherkg = 0 or G is k-hyperbolic. 


Proof. In the case when Gis c-hyperbolic there is the holomorphic covering 7 : D —+ G. According to Remark 
7.2.24(a) there exist points ',rA” in the unit disc such that IW(\’)=2', I(A")=2z", and 
ke(z',2") =kpQ’,A") = pQ’,X”). On the other hand, cg¢(z’,z”) can be written as 
cg(z’, 2”) = p(f(z’), f(z") for a suitable f € O(G,D). For the function f° IT € G(D, D) this implies 


p(f°M(\'), f° H(A")) = ec(2’, 2") = ke(z’, 2”) = p(X’, A"). 


Now, the Schwarz—Pick lemma tells us that f° JZ is a biholomorphic map, and therefore I is biholomorphic. 
We turn to the proof of claim (b). Since Gis not c-hyperbolic, we have cg = 0. In the case where the universal 


covering of Gis given by ©, it is clear that kg = 0. So we may assume that JJ : D — G is the universal covering. 
Hence by Theorem 7.2.23 and Remark 7.2.24(a), we conclude that whenever z,2"€G,2 x z'', then there are 
points 2’, A” € D, (A’) = 2’, IA") = 2” with kg(z’, 2”) = p(X’, X”) > 0. 


Corollary 7.2.26. Let P:={X:1/R<|A|< R} (R>1 ). Then for z, 2" EP, 2 #2", we have 
cp(z', 2") <kp(z’, 2”). 


7.2.5 The Kobayashi-Royden pseudometric 


In Chapter 7.1 we have already learned that for the Carathéodory pseudodistance there is an infinitesimal version, 
the Carathéodory—Reiffen pseudometric, which measures the lengths of tangent vectors. A similar notion with 
respect to the Kobayashi pseudodistance was introduced by Royden in 1997 (see [Roy71]). Because of its strong 
relation to the Kobayashi pseudometric it will be introduced here but only shortly investigated. 

Let G be a domain in C*. The = function kKe:GxC"*>R, defined by 
Ko(z; X) := inf {y(A)|a| : dyeow,e) drew: Y(A) = 2, ay’(A) =X} is called the Kobayashi-Royden 
pseudometric. 


Remark 7.2.27. Observe that: 


Q) K@(z;X) = inf {a > 0: dye.) : (0) = z, ay’(0) = X} 
= inf {a > 0: Ace)? (0) = 2, ay'(0) = xh; 
(2) K@(z;AX) = |A|Ke(z;X), ACC, X EC, zEG CC; 
(3) Kp(F(z); F'(z)X) < K¢(z;X), F € G(G,D), z€GcC", XEC". 


Hence Kg(z; -) assigns a length to any tangent vector at z, and moreover (3) shows that the system (Kg) ¢ 
is contractible with respect to holomorphic mappings. In particular, if F : G — D is a biholomorphic map, 
then 


(4) Kp(F(z); F’(z)X) =Ke(z;X), EG, X EC"; 
(5) Ko(;X) = y()|X] = yo(z; X), A€ D, X EC; 


(6) for any domain G C C” we have y@ < Kg (use the Schwarz—Pick lemma); 


(7) for any domain G C C” we have yq < Keg; 
(8) «gis upper semicontinuous on G x C”; note that, in general, «Gg is not continuous (see Proposition 7.2.29); 
(9) Ke@,xG,(a;X) = max{Kg, (a1; X1),KG,(a2; X2)}, where G; C C™, j = 1, 2, are domains and 

a = (a1, a2) € Gy x Gp and X = (Xi, X2) € C™ x C™, ie. w satisfies the product property. 


Moreover, it turns out that whenever there is a system (dg) of functions dg : G x C" > R, (GC C”) with 
the properties (3) and (5), then dg < Kg for any G. We recall that we already know that also yg < dg is true. 


Remark 7.2.28. Recall that yg could be thought as a certain derivative of the Carathéodory pseudodistance (cf. 
Remark 7.1.8(d)). A similar result is true with respect to the Lempert function. 


(a) Assume that G C C” is a taut domain. Then (see [Pan94], [NP08]) 


Kg(a;X) = lg(z,z+AX'), aeG,xXeEC". 


lim + 
c\{o}sA30 
za 
X'S X 


(b) Without the assumption that Gis taut this formula remains, in general, not true. 


To see at least a few concrete examples, we calculate the Kobayashi-Royden pseudometric for balanced 
pseudoconvex domains. 


Proposition 7.2.29. Let G = Gy be a balanced pseudoconvex domain in C” given by its Minkowski function h. 
Then kg(0;X) = §(X), X € C”. 


Proof. First of all observe that if h(X) #0, then DDA ne AX /§(X) is an analytic disc in G with y(0) = 0, 
yp’ (0) = X/h(X); hence we obtain kg(0;.X) < h(X). 

The fact that the same inequality is also true if h(X) = 0 is left to the reader (cf. Example 7.2.16). On the other 
hand, let y € 6(D,G) with y(0)=0, ay'(0) =X (a>0). As in Proposition 7.2.5, we observe that 
h(y(A)) = 6(AG(A)) < 1, and therefore °G < 1. 

Thus we end up with h(X) = h(ay’(0)) = ah* G(0) < a, which guarantees the missing inequality. 


We emphasize that the proof above is based on the information that G is pseudoconvex, i.e. that the 
Minkowski function is plurisubharmonic; in general it is false (cf. [FS87]). 
Applying Proposition 7.2.29 leads to the following formulae. 


Example 7.2.30. 
(a) For z € B, and X € C”, the following formula is true. 


x1)? 


Kp, (2; X) = (sar 


For the proof use the fact that kg, is invariant under Aut(B,.). 

In particular, this formula immediately yields to the following observation: if G is a bounded domain, then 
there exists a constant C' > 0 such that for any z € G, X € C” the following inequality Kg(z;X) > C||X|| is 
true. 


(b) A similar argument leads to the corresponding formula for the unit polycylinder. 


1-|z|* ieee} 1-|z,|? 


Kpe(%X) = max{ es sacg ee \ 


(c) Moreover it turns out that, in general, Kg(z;-) is not a norm on C’. For example put 
G:= {z €D?: |z12z2| < 1/2}; G is a bounded balanced pseudoconvex domain with Minkowski function 


h(z) = max |21 , zal, V2]z1| zal}. Therefore we know that Kg(0;X)=6(X). In _ particular, 
Ko (0; (3,3)) =% > 2 = Ka (0; (1,2)) + Ka(0; (2, 1). 
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Moreover, Proposition 7.2.29 can be also used to describe biholomorphic mappings between balanced 
pseudoconvex domains. 


Corollary 7.2.31. Let G; = Gy,£C” be pseudoconvex balanced domains with Minkowski functions h;, j = 1, 2. 
Then the following conditions are equivalent: 


(i) there exists a biholomorphic mapping F : G, > G» with F(0) =0; 


(i) there exists a C-linear isomorphism L : C" — C” such that hy° L = hi, ie. G; and G; are linearly 
equivalent. 


Remark 7.2.32. 

(a) Observe that W:=— F-1°F ‘(0) is a biholomorphic mapping of G, sending the origin to the origin. Moreover, 
W'(0) = id. In case where G, is bounded, a result similar to Lemma 7.1.46 finally yields that ~ = idg, or 
F = F'(0), ie. Fis a linear isomorphism. Note that the assumption “G, is bounded” is essential-take for instance 
G, = Gp = Cand Y(z) := z(1— 2). 

(b) Moreover, a deep result by Kaup—Upmeier (see [KU76]) says that whenever there exists a biholomorphic 
mapping F':G,— G2, then there is another one sending 0€ G; to 0€ Gp. In particular, if G, is 


biholomorphically equivalent to G,, then G, is linearly equivalent to G,. 
Similarly as in the case of the Lempert function (cf. §7.2.3), we also obtain better results for the Kobayashi— 
Royden pseudometric on taut domains. Since the argument here is more or less the same, we only formulate the 


result. 


Proposition 7.2.33. Let G be a taut domain in C”. Then 


(a) for any z € G and for any X € C” there exists an extremal analytic disc p € G(D, G), i.e. g(0) = z and 
Kq(%X)¢'(0) =X; 


(b) the Kobayashi-Royden pseudometric is continuous on G x C”. 


Recall that the y¢—length has led to the inner Carathéodory pseudometric which, in general, is different to cg. 


On the other hand the Kobayashi pseudodistance can be obtained via integration of the Kobayashi-Royden 
pseudometric. 


Theorem 7.2.34. Let G C C” be an arbitrary domain. Then 


ke(z', 2") = inf{L,,(a) : a a piecewise C!-curve in G connecting z' and z”}, 


where L,,(a) = | Kg(a(t); a'(t))dt. 


As a direct consequence of Remark 7.2.27(9) and the former theorem one obtains the product property for the 
Kobayashi pseudometric. 


Corollary 7.2.35. LetG; C C™, j = 1,2, be given domains. Then 
ke, x@, (a, b) == max{kg, (a1, bi), kg, (a2, b2)}, 
where a = (a1, @2), b = (b1, bg) € Gi x Go. 


7.2.6 k-hyperbolicity 


Let M and N be complex manifolds with M a relatively compact open subset of N, and f:D,—> M a 


holomorphic mapping. When does f extend to a holomorphic mapping fi D — N ? The classical case of this 
question is when NV = P? is the Riemann sphere and M = P1\{0, 1, oo}. Then extending fis equivalent to the big 
Picard theorem. Recall that MM here is  khyperbolic. On the other hand put 


M := {(z, w)€D,xC: |w| < Ve < i} C P?. Then (see [Kie70]) M is khyperbolic. (Using similar 
techniques, it can be shown that M is complete hyperbolic). Define f : D, + M putting f(z) := (z,e!/7). It is 


easy to see that fcannot be extended to a holomorphic map f: D —> P?. Hence, the big Picard theorem cannot be 
simply generalized to higher dimensions. According to this extension problem there is some need to discuss & 
hyperbolicity. 

The following generalization of the big Picard theorem is given by M. H. Kwack (cf. [Kwa69]). 


Theorem 7.2.36. Let f :D. — G be a holomorphic map, where G is a k-hyperbolic domain in €”. Assume that 
for a sequence (Ax), C Ds with 4x — 0 the sequence (f(Ax))-., converges to a point z) € G. Then f extends 


to a holomorphic map f: > G. 


Remark 7.2.37. Take G := C\D and f : D, — G, f(z) :=1/z. Then Gis hyperbolic, but fdoes not extend to the 
whole unit disc. Therefore, it is clear that an additional property of fis needed to get the extension result. On the 
other hand there is the following example by D.D. Thai and P.J. Thomas (see [TT98]): there exists a subharmonic 


function u on D, such that the Hartogs domain H, := {z eDxC: |z2| < ela) } is not Kobayashi hyperbolic, 


nevertheless any holomorphic f € @(D,, H,,) extends to a holomorphic map f : D > Hy. 


Proof of Theorem 7.2.36. Without loss of generality, we may assume that the sequence rz := |A;| is strictly 


decreasing and that z) = 0 € G. We are going to prove that the function f given by fA) := f(A), AE D,, 
f(0) := 0 is continuous on D. Fix an € € (0, dist(0, 9G)). Since G is khyperbolic, we have By,,(0,6-) C B(e) 
with 6, being a suitable positive number. By assumption there is a kg € N such that for k > ko we have 
f(An) € Bx, (0, 6-/2). Applying Remark 7.2.24(g) for A € D., |A| = rx, we obtain 


kg (0, f(A)) < kg(0, f(Ak)) + ke(f (Ak), F(A)) 
<ko(0, f(Ax)) + kp, (Ag, A) < 2% + —2— < 6, if k > hy > ko. 


2 —logr, 


Therefore, f(OD(r;)) C Ble) if k > ky. 

It remains to show that if rz4, < |A| < rz, k > 1, then f(A) € B(e). But this is an easy consequence of the 
fact that the Euclidean norm is a plurisubharmonic function and the maximum principle for the subharmonic 
function A +> || f(A)||. 


Hence, it is important to have tools to decide whether a given domain is hyperbolic. 
In the class of convex domains containing the origin the following result due to T. J. Barth (cf. [Bar80]) is true. 


Theorem 7.2.38. Let G C C” be a convex domain, 0 € G. Moreover, assume that no complex line through 0 stays 
inside G. Then G is biholomorphically equivalent to a bounded domain. In particular, G is k-hyperbolic. 


Remark 7.2.39. 
(a) It seems to be still an open problem whether Gis even biholomorphic equivalent to a bounded convex domain. 
(b) In [BS09] one may find even a longer list of equivalent properties for a convex domain to be Khyperbolic. 


Similar results for C-convex domains are given in [NSO7]. 


Moreover, the following criteria for hyperbolicity are extremely useful. 


Theorem 7.2.40. Let IT : G > G be a holomorphic covering. Then we have: G is k-hyperbolic if and only if G is 
k-hyperbolic. 


A similar result is due to A. Eastwood (see [Eas75]). 


Theorem 7.2.41. LetG,; C C”, j = 1,2, be domains and let F € O(G1, G2). Assume that G, is k-hyperbolic and 
that there exists an open covering (Ua) 4-4 Of Gz such that each connected component of F~\(U,) is a k- 
hyperbolic domain, a € A. Then G, is k-hyperbolic. 


Since it is simpler to deal with #g than with Kg, it is important to express the property “hyperbolic” in terms of 


the associated metric. Let G be any domain in €”. Then G is x-hyperbolic if for any z € G there exist a 
neighborhood U = U(zo) C G and a positive real number C such that Kg(z;X) > C||X||,z €U, X © C”. 
There is the following characterization of Khyperbolicity. 


Theorem 7.2.42. For a domain G in C" the following properties are equivalent: 


(i) Gis k-hyperbolic; 
(ii) topG = topkg ; 
(iii) for any domain G’ Cc C™, any w' € G’, any z' © G, and any neighborhood U = U(z') C G there exist 


neighborhoods V = V(w') C G' andU = U(z') CU such that if f € 6(G',G) with f(w') € U, then 
fV) CU; 


(iv) condition (iii) is true forG' =D andw' =0€D; 


(v) Gis x«-hyperbolic; 
(vi) for any z' € G there exists a Kobayashi-ball around z with finite radius r, which is a bounded subset of C"; 


(vii) any point z' € G has a neighborhood U = U(z') C G such that, for z, w € U,kg¢(z,w) > M||z— wl, 
where M is a suitable positive constant. 


7.2.7 Examples 


In the class of Reinhardt domains a complete characterization of hyperbolicity exists. To be able to formulate this 
theorem several notions are needed: 

eVp=H={2eCig=0, goles 

© Dac := {2 €C", 2; £0 if ay <0: |z1 1... len On el. where a € R”\{0} and C € R; 

« M(n x n; S$) := the set of all n x n-matrices with entries in S C C; 

- for A= (AL) ee x € M(n x n;Z), denote by A/ its j-th row; 


- for A € M(n x n;Z) put 4 : (C\{0})” — (C\{0})”, H(z) := (24’,..., 24”). 


Theorem 7.2.43. (cf. [Zwo99]) Let G be a pseudoconvex Reinhardt domain in C”. Then the following properties 
are equivalent: 

(i) Gis c-hyperbolic; 

(ii) Gis k-hyperbolic; 


(iii) logG := {a € R”: (e”!,...,e7") © G} contains no affine lines, and either VV; G = 9 or V;.G (treated 
as a domain inC”~ ) is c-hyperbolic, j =1,...,n ; 


(iv) there exist A € M(n x n;Z), |det A| = 1, anda vector C € R” such that 
GC Dac = Dac, (Vs.2 20 Da-c,, 


eeither V; 0G = 0 or Vj G is c-hyperbolic as a domain inC"™, j =1,...,n; 


(v) Gis algebraically equivalent to a bounded domain (i.e. there exists a matrix A € M(n x n;Z) such that © 4 is 
defined on G and gives a biholomorphic mapping from G to the bounded domain © 4(G) ); 


(vi) Gis k-complete, (i.e. G is k-hyperbolic and every kg-Cauchy sequence in G converges (in the standard 
topology) to a point in G). 
7.2.8 Kobayashi completeness 


Since ec < k, every ecomplete domain is kcomplete. So Section 7.1.8 provides a lot of examples of kcomplete 
domains. On the other hand, the following necessary condition shows that there are many domains which are not 
kcomplete. 


Proposition 7.2.44. Any k-complete domain is taut. 


Proof. Let (p;);-, C @(D,G). Assume that (y;);", is not uniformly divergent. This implies that there are 
compact sets K C D and L C G such that, without loss of generality, y;(A;) € L with A; € K. Fix z* € L and 
let 0 <r <1 with K C D(r). Then for A € D(r) we obtain 


ke(9;(A),2*) < ke(y;(A), 53) + ke (H5(A3), 2") 
< p(A, A;) + sup{ke(z, 2*): ze L} =: C. 


Hence, U pil (r)) C Byg(z*,C +1) CC G. Therefore, Montel's theorem guarantees the existence of a 
je 


subsequence (yj,)>~, C (95) joa which converges locally uniformly to a map in @(D, G)). 


Corollary 7.2.45. Any k-complete domain is a domain of holomorphy. 


Remark 7.2.46. For a while there was the question whether tautness can imply Kcompleteness. The first negative 
example was found by J.-P. Rosay (cf. [Ros82]). Later in this section we will present another example. 


There is a simple example of a domain which is not c-complete, but which is complete, namely the punctured 
disc D,. This observation is a direct consequence of the next result due to S Kobayashi (cf. [Kob67], [Kob05]). 


Theorem 7.2.47. If II: G —> G denotes a holomorphic covering between domains in C", then the following 
statements are equivalent: 


@ Gis k-complete; 
(ii) Gis k-complete. 


Proof. (i) = (ii): According to Theorem 7.2.40, Gis k-hyperbolic. Fix a ball By, (Zo, 7) in G. By Theorem 7.2.23 
it is clear that By,(z0,7) C II(Bx,(Z0,7)), where 2p is a point in G with IT(Z9) = zo. Recall that kg is inner and 
so this implies that kcompleteness is equivalent to K finite compactness. Hence, By; (20,7) CC G, and therefore 
Bx, (Zo, 7) CC G. 

(ii) = (i): As above, G is khyperbolic. Fix ak g@-Cauchy sequence (2,)° ) C G. Then obviously (II(z,))°, 
is a k,-Cauchy sequence. By assumption, this sequence converges to a point z) € G. Using again Theorem 7.2.23 


it is easy to construct a subsequence (2,,) of (z,)%°, and points Zo, ¢G with I(Z,,) = zo and 


p=1 
ka (21,20) < 1/u. Thus ka (Zo,,,; 20,a) wae 0. On the other hand, there exist a neighborhood By,(z0, 17) of % 


and neighborhoods U, of Zo, such that IZ|y,:U,— Bye(zo,7) is biholomorphic,  ¢N. Put 
V, = I~! (Bx,(20,7/2)) NU ,; then kg (Zo, 0V,) > 7/2. This observation together with k, = k’ shows that 


for a sufficiently large up we obtain Zo, = Zo, > Mo. Put 2 := Zo,,. Then a standard argument leads to 


Example. C\{0, 1} is a &complete domain but it is even not c-hyperbolic. 


We will see that the property of k-completeness is a local one (cf. [Eas75]) in contrast to the property of being 
c-finitely compact. 


Theorem 7.2.48. Let G be a bounded domain in C”. Suppose that any boundary point z) € OG permits a bounded 
neighborhood U = U(zo) such that any connected component of GU is k-complete. Then G itself is k- 
complete. 


Remark 7.2.49. The above theorem also provides a simple argument to show that any strongly pseudoconvex 
domain is complete using only the existence of local peak functions. 


By Theorem 7.2.43 we know that any bounded pseudoconvex Reinhardt domain containing 0 is kcomplete. 
Moreover, tautness, i.e. the continuity of its Minkowski function, is necessary for a balanced domain to be & 
complete. Nevertheless, the following result shows that tautness, even in the case of a balanced domain, does not 
imply &completeness. 


Theorem 7.2.50. (cf. [JP9la]) There exists a bounded taut balanced pseudoconvex domain 
G = Gy, = {z € C”: h(z) < 1} = 3) with continuous Minkowski function b which is not k-complete. 


Remark 7.2.51. 


(a) [?] It would be very interesting to know whether such an example could be also constructed in @ [?] We 
emphasize that the method used to prove Theorem 7.2.50 does not work in the two-dimensional case. 


(b) [?] So far it is totally unclear how to characterize the kKcompleteness (or the c-completeness) of a bounded 


pseudoconvex balanced domain via the properties of its Minkowski function [?] 


Up to now [?] it is an open problem whether every bounded pseudoconvex domain with @°°-smooth boundary 


is kcomplete The strongest result in the negative direction is the following unpublished one due to N. Sibony; 
cf. [Sib8 La]. 


Theorem 7.2.52. There exists a pseudoconvex non k-complete domain G CC Bg given as a connected component 
of {z € B(3) : u(z) < 1}, where u € PSH(B(3)) N@(B(3)) n@~(B(3)\{0}), gradu(z) £0 if z #0, and 
u(0) =1. 


To conclude this section one should emphasize that kcompleteness is much easier to handle than c 
completeness; nevertheless, there are still a lot of unsolved questions. 


De 
7.3 Lempert's theorems and the symmetrized bidisc 


In this last chapter, we discuss various results telling that for certain domains G C C” the extremal invariant 
pseudodistances coincide. 


7.3.1 Lempert's theorems 


Theorem 7.3.1. (1%* theorem of Lempert, [Lem81], [Lem82]). If G Cc C” is a domain which is the union of an 
increasing sequence of domains G; each biholomorphically equivalent to a convex domain, then eg = Lg. In 


particular, the Carathéodory pseudodistance and the Kobayashi pseudodistance coincide on such a G. 


To give an idea of the proof one first observes that it suffices to prove Theorem 7.3.1 in the case when G is 
bounded, convex, and 0 € G (cf. Remark 7.1.8(c) and Proposition 7.2.8). 


So from now on G is assumed to be bounded convex and 0 € G. Fix then two different points a and b in G. 
Then Proposition 7.2.11 allows to find an extremal disc y € O(D,G) with y(0)=a, y(o) =b, and 
p(0,c) = g(a, b). So it remains to find a holomorphic function f € G(G,D) such that f°y = idp. Indeed, 
P(r’, A”) < eg(e(A’), P(A") < Le(9(’), P(A") < P(A’, A"), A’, A” € D; in particular, eg(a, b) = £¢(a, 6). 

To see how convexity enters the proof a few notions have to be repeated. Let 
zew:= (2, Ww) =7zjU,; +...+ Z,Wn. Since G is convex, there is the Minkowski function 
ga(z) := inf{t € (0,00) : z/t € G} and the dual Minkowski subnorm gg¢(z) = sup{Re(ze w) : w € AG}, 
z € C”. Note that a point z sits in Gif and only if gg(z) < 1. 

Let F := C(T,C"). Then F together with the norm || f|| 7 := sup{|f(A)| : A € T} is a Banach space. If py € F 
, put Q(¢%) :=sup{gqe((A)):A€¢T}. Then Q is a Minkowski subnorm on EF Moreover, let 


A: {v eF:peC,c")n o(D,c")}. 

Now fix an arbitrary analytic disc gy) € O(D,C") with gyo(A’)= 2’, Yo(A”) =z” and define 
Vo := {W € A: YA’) = YA") = 0}. According to Remark 7.2.12(b) it follows for any w«€ Vo_ that 
sup{qa(po(A) + H(A)) : A € D} > 1. Consequently, Q(yo + W) > 1. 

Looking at the functional Ryo + Vodtwo + w er t it is easily seen that 1 < Q and so, by the Hahn-Banach 
theorem, it can be extended to a R-linear functional L on F, L < Q. Finally, setting u() := L(iw) — iL (iw) one 
ends up with a continuous linear functional ~ € F'’ with july, = 0; u may be thought as a Borel measure on T. 


Using the F. and M. Riesz theorem one concludes that wis given as pp = ato 


mapping from D to C’, each of its components belongs to the Hardy space on hEH a | ,C”)), the star means 


= 


the boundary value componentwise, and 7 stands for the normalized Lebesgue measure on T. 
Put w:= ENVOY on \{\',A"}. Then it~ follows = that = gg(y*(A)) =0 and 
Re(y*(A) e u*(A)) = Ge(u*(A)) almost everywhere on T. Modifying h leads finally to a mapping 


h € H'(D,C"), h £0, such that Re(y*() e ay) = Go(7) for almost all A € T. So far the geometric 


property has been heavily exploited. 


,A\€ T, where h is a 


The final step starts with the condition that Re((z— y*(A)) e a) <0 for z€G and almost all A € T 


which is a consequence of the result just before. Then pure complex analysis reasoning leads to the existence of the 
left inverse for g which was above postulated to be found. 
The proof above is based on the unpublished paper [RW83]; details may be found in [JP13]. 


Remark 7.3.2. To generalize the above situation, an analytic disc p € 6(D,G), Ga domain in C, is said to be 
weak m-extremal for points A1,...,Am © (pairwise different and m => 2) if there is no analytic disc 
h € 6(D,G) with p(A,) = A(x), k= 1,...,m. And a mapping f € @(D,G) is called an m-extremal mapif 
there exists a holomorphic mapping F' € @(G,D) with F’° f = B, where B is a Blaschke product of order m — 1. 
So Lempert's result as described before may be read as: any weak 2-extremal disc is a 2-geodesic. There was some 
hope that the Lempert result may be generalized in this context claiming that if G is convex, then any weak m- 
extremal analytic disc p € @(D, G) is also m-geodesic for all m > 2. But as it turned out such a proposition is 
already false for G = Bz and m > 4 (see [KZ16]). Moreover, there are convex domains for which this kind of 
result fails to hold for m = 3 (cf. [War15]). 


To be able to formulate the second result the following notion is needed. A domain G C C” with a @?- 
“. Or 
0z;0Z% 


defining function r is called to be strongly linearly convexif (2r)(a;X) > (a)X;X;|, a € OG, 


j,k=1 
X € TE(dG)\ {0}. 


Theorem 7.3.3. (2°¢ Lempert theorem, [Lem84]). Let G C C” be a domain which can be written as the union of 
an increasing sequence of bounded strongly linearly convex domains G,. Then again ¢g = a. 


A readable and complete proof can be found in [KW12]. 


Later, D. Jacquet (see [Jac06]) was able to extend the above results to bounded €-convex domains with a 
smooth C? boundary. 


Theorem 7.3.4. Let GC C” be a bounded C-convex domain with a smooth @? -boundary. Then G can be 
exhausted by a sequence of bounded strongly linearly convex domains; in particular, cg = fg. 


A few words to the proof of Jacquet's result: 


- A function u € @?(D; R), Da domain in C, is called to be Cconvex (resp. strongly Cconvex), if 


2 n 
Lul(a;X) > a (a) Xj; Xz — SS ax ; 


(resp.>) | G4 0z;0Z% =I Oz; 


aé D, X €C"\{0}. 


« Let D C C” be a domain with @?-boundary. Then: 
Dis C-convex if and only if — log dist?(-, OD) is a C-convex function on DM U for some open neighborhood 
U =U(OD). 

- If uis a Gconvex function, then u, := —log(e” — e(1 + ||z||”)) \.u as € \, 0, and the u, are strongly GC 
convex functions. In particular, this leads to an exhaustion by strongly linearly convex domains. 


At the moment it is unclear whether the former theorem remains true without assumptions on the smoothness 
of the boundary. 


7.3.2 The symmetrized bidisc 


For a long time it was absolutely unclear whether Theorem 7.3.3 is eventually a consequence of Theorem 7.3.1. 
The surprising answer to that question was found by discussions which do not belong directly to the field of 
several complex variables; it stems from discussions of the so called y-synthesis problem. That is an interpolation 
problem for analytic matrix valued functions which may be thought as a generalization of the classical problems of 
Nevanlinna—Pick. Here uw denotes a positive cost function that generalizes for example the operator norm for 
matrices. Then the y synthesis problem is to find an analytic matrix valued function on D such that it satisfies a 
finite number of interpolation conditions together with the condition that 44(f) < 1 on D. The precise definition of 
the cost function y will be omitted; for our purpose pis taken as the spectral radius r(A) of a2 x 2 matrix A. 

In this section as a special case of the y-synthesis problem only the spectral Nevanlinna—Pick problem will be 
discussed. To be more precise: given pairwise different points A,,..., Ay € D and k x k matrices A,,..., Ay, 
r(A,;) < 1, then the problem is to construct an analytic k x k matrix valued function Fon D such that F(A;) = A; 
,j=1,...,N, and r(F(A)) < 1 for all \ € D, where r(A) := max{|A| : A is an eigenvalue of A} denotes the 
spectral radius of A. 

When k = 1 this is just the classical Nevanlinna—Pick problem, and it is well known that a suitable F exists if 
and only if a certain n x n matrix formed from the Ajs and A/s is positive definite (this is Pick's Theorem). 

Put \varOmega, := {A € M(k x k;C) : r(A) < 1} and call this set in C* the unit spectral ball. So the 
above problem consists in finding certain holomorphic mappings into \varOmega, with prescribed values at 
given points. 

The unit spectral ball shares a lot of interesting properties. Only a few will be mentioned here: 


- \varOmega, is a pseudoconvex domain but it is neither convex nor bounded (in fact it contains even complex 
lines); 
- \varOmega, is balanced, but not taut (i.e. the Montel theorem does not hold for @(D, \varOmega,)). 


k 
Recall that if A € \varOmega,, then det(xI, — A) = x" + > (—1)’2b,,;(A)a* 4, where the (A) is 
j=l 


given via the elementary symmetric polynomials o;,; and the eigenvalues of A. Hence, there are the following 
maps 7, : \varOmega, > C*, At> (We1(A),---,Pee(A)) and ox : DF > C*, Xs (041 (A),---, OR K(A))- 


Then it turns out that ~;(\varOmega,) = o%(D*) =: G;, is a domain in C which is called the k-dimensional 
symmetrized polydisc. 

If the spectral Nevanlinna—Pick problem with data A1,...,Aw € D and Aj,..., Aw € \varOmega, can be 
solved, then using y;, the interpolation problem with data 1,...,Ay € D and ,(Aj),...,%%(An)) € Gy 


allows a solution in G,. So the original problem may be studied in two steps: 


(a) solve the corresponding interpolation problem in G, (note that the number of parameters now involved is 
N(1+-k), while in the spectral Nevanlinna—Pick problem it is N(1 + k?); 
(b) try to lift the solution in G, to a holomorphic mapping F from D to \varOmega, with F(A;) = Aj. 
From now on it is assumed that k = 2, N = 2. Then G, is the so called symmetrized bidiscwhich shares the 
following interesting geometric properties most of them are found in [AY04]: 
« w2(A) = (trace(A), det(A)); 
- G= {(s, p) EC?: |s — 5p| + \p|? = 1}; in particular, G, is bounded, pseudoconvex, and taut; 
+ G, is not convex and has no smooth boundary; 
¢ G, cannot be exhausted by domains G; biholomorphic to convex domains (see [Cos04b], [Cos04a], [Edi03]); 
i.e. Theorem 7.3.1 cannot be applied; 
¢ G, is C-convex (see [NPZ08]); 
+ G, is not homogeneous, the orbit of 0 under Aut(G2) is given by {(2A, A”) : A € D}. 


In order to study the spectral Nevanlinna—Pick problem one has to find conditions to solve the corresponding 
interpolation problem in the symmetrized bidisc. So let A1, A2 € D and points 2’, z” € G2 be given. Without loss 
of generality one may assume that z’ 4 z” and A; = 0 and Ay = a € (0,1). Obviously, a necessary condition for 
the existence of a solution is that £4, (z', 2) <o. On the other hand, if ¢%,(z', 2”) <o, then there exists an 
analytic disc y € O(D, G2) with y(0) = 2’ and y(r) = z” with a certain 7 € [€g,(2’, 2”), 0]. Then the disc 
@(A) = v(AT/c) does the job of interpolation. So it remains to determine the number £%,(z’, 2”) or at least a 
lower bound. Note that a first lower bound is obviously given by the Mobius function mg, (z’, z’’). 

In fact, J. Agler -N. Young [AYO1] and C. Costara [Cos04b], [Cos04a] were able to prove the following result. 


Theorem 7.3.5. (Theorem of Agler-Young, Costara). 


mg,(z’, 2”) = max{m(#)(z’),&)(z”)) : A € T} = ee, (2, 2"), 


where ®, : G2 > D, (s,p) op . In particular, €g, is given by the following concrete formula 


(s1p2—s2p1)¢?+2(pi—p2)¢-+s2—81 
(S2p1—$1)¢?+2(1—pip2)¢+s1p2—52 |" 


Max(cT 


An different proof was given by P. Pflug and W. Zwonek (see [PZ12]). 


Theorem 7.3.6. The symmetrized bidisc can be exhausted by bounded strongly linearly convex domains. In 
particular, by Theorem 7.3.3, cg, = lc). 


As a byproduct this result together with former properties of G, leads to the following consequence answering 
an old question by L. Aizenberg. 


Corollary 7.3.7. There exist a strongly linearly convex domain in ©? which cannot be exhausted by a sequence of 
domains biholomorphically equivalent to convex domains. 


The first idea to extend the above result to higher dimensions was without success. In fact, in [NPZ07] and 
[NPTZO08] the authors were able to show that the equality of cg, and fg, fails for m > 3. Nevertheless, there is still 


a great hope that this equality remains true for all C-convex domains. 


Remark 7.3.8. It should be mentioned that the symmetrized polydisc appears as a special case of a large family of 
other domains whose construction will be shortly presented (see [Zap15]). Fix natural numbers n > 2, s < n, and 


T1,++-)Ts5 Such that > r; =n. On the product space A := Il {0,1,... a nes a} define an ordering 


j=l j=l 
saying for a, 6 € A that a < # if there exists an jo € {1,...,s} with aj, < Bj, and a; = 6; for jp <j <s. 


Then A = oe eek a for N := IIe +1) +1 where aJ < aJ*? for all possible j. Then the following set 
j=l 


el 
I 


N _ 
Danita {* €C¥:14+ 50 (1 ler” 4 Ofor all Ac i 


j=l 


is a bounded pseudoconvex domain satisfying the following properties: 


« E cannot be exhausted by domains biholomorphically equivalent to convex ones; 


+ if one of the r; > 3, then cg ¥ &g and E is not C-convex; 
e if rg =---rs = 1, then E is linearly convex and, consequently, pseudoconvex. 


Moreover, E2111 = Ge and for the so-called tetrablock E221,1 (see [AW Y07]) the Carathéodory pseudodistance 


and Lempert function coincide and both domains are C-convex (see [EZ09], [EKZ13], and [Zwo13]). 
So the discussion of this general class of domains gives hope that C-convexity may lead to the equality of the 
Carathéodory pseudodistance and the Lempert function. 


What remains is to answer problem (b) of the possibility to lift solutions of the Nevanlinna—Pick problem to 
solutions of the spectral Nevanlinna-Pick problem (see [NPT11)). 


Theorem 7.3.9. Let A, = aylz,..., Ag =axl2 € \varOmega,, Agy1,..., Ae € \varOmega, non scalar 
matrices (i.e. matrices which have not the form of the previous Aj), and p € O(D, Gz) such that (Aj) = p(Ay), 


j=1,...,£. Then there exists a G € O(D, \varOmega.,) satisfying =) G = y and G(A;) = A; forj =1,...,2, 
if and only if p4(A;) = ajp{(A;), 7 = 1,...,k. 


Hence, the spectral Nevanlinna—Pick problem is solved in dimension 2 and, as a byproduct, new information 
with respect to the equality of c and @ is obtained. Nevertheless, as it was seen a lot of unsolved problems remain 
for further research. Let us finish this article with the one of the most important conjecture in this area: Let 
G C C” be C-convex, then eg = ég. All experiences so far make it highly probable that the answer will be “yes”. 


7.4 Epilogue 


7.4.1 The Green function 


First let us recall the notion of the classical Green function (cf. [Ran95]). Let G C C be a domain and let a € G. 
We say that a function g¢(a,-) : G — (—00, +00] is the classical Green function of G with pole at a if: 


* ga(a, -) is harmonic on G\{a}; 
* gg(a,a) = +00; 
+ the function G\{a}5z +> gc¢(a, z) + log |z — a| extends to a harmonic function on G; 
- there exists a polar set F C OG +, such that: 
— if¢ € (OG)\F, then lim gg¢(a, z) =0, 
Gaz>¢ 


- if¢ € For¢ = o € OG, then g¢(a, -) is bounded near C. 


44 set A C Cis said to be polar, if for every a € A there exist r > 0 and u € SH (D(a, r)), u % —oo, such that u = —oo on AN D(a, r). 


Observe that gp(a, z) = — log | 


,a,Z€ 


Theorem 7.4.1. (a) IfG C C is a domain such that OG is not polar, then for every a € G the Green function 
9c(a,-) exists and is unique. Moreover, gg(a,z) > 0 for z#a and the function gg : G x G — (0,+00] is 
symmetric and continuous. 

(b)If ga(a, -) exists, then, by the maximum principle for subharmonic functions, we get 


—~ga(a, z) = sup{v(z) 1VvE SHG, [—00, 0)), 


sup (v(w) — log 
weG\{a} 


W—-a 


)< 40h, zeG. 


For an arbitrary domain G C C”, Theorem 7.4.1(b) suggests the following definition of the pluricomplex 
Green function with pole at a: 


ga(a, z) := sup{u(z):u:G— [0,1), logu € PSH(G), 


sup ay < +00 , zeG. 
weG\{a} 


With this notation Theorem 7.4.1(b) reads as follows: forG C C, if gg(a, -) exists, then gg(a, -) = — log g(a, -). 
Observe that gp = m and for any holomorphic mapping F': G++ D we have gp(F(z), F(w)) < gc(z,w), 
z,w © G. Thus the system (gq) is holomorphically contractible and therefore mg < gq < £6. 

Similar to cg and fg, the  pluricomplex Green function has its infinitesimal form 


Ac(a; X) := limsup ppeo(a,a + AX), aéG, X €C", which is called the Azukawa pseudometric (cf. 
A—0 
[Azu86]). One can easily check that Ag(a;AX)=|A|Ac(a;X), Ap(a;1)=y, and 
A p(F(z); F'(z)X) < Ac(z;X) for any F € 6(G, D). In particular, yg < Ag < kg. 
The following deep result corresponds to the harmonicity of g¢(a,-) on G\{a}. It establishes an important 
link between the theory of the pluricomplex Green function and pluricomplex analysis. 


Theorem 7.4.2. (cf. [Kli85]) For each a € G the function log gg(a,-) is a maximal plurisubharmonic function on 


G\{a}. In particular, if log gg(a,-) € L¥,(G\{a}) (eg. if the set G is bounded), then (dd° log gc(a,-))” = 0 in 
5 


G\{a}, where (dd°-)” denotes the Monge-Ampére operator °. 
We collect below various basic properties of the Green and Azukawa functions. 


Remark 7.4.3 (cf. [JP13], Proposition 7.2.10) 


(a) logg¢(a;-) € PSH(G) and sup 22%) < +00. 
wea\{a} al 


(b) log Ag(a;-) € PSH(G). 
(c) IfG, AG, then gg, \. gg and Ag, \ Ag. 


(d) If G = G, C C” is a balanced domain, then the following conditions are equivalent: 


(i) ga(0,-) =b onG; 
Gi) Ag(0;-) = hon @; 
(iii) Gis pseudoconvex. 
(e) If Gis bounded, then for any zp € G the function GDz +> ge(z, Zo) is continuous. For unbounded domains 
G the function GDz +> ge¢(z, zo) need not be continuous; cf. Example 7.4.4. 
(f) For any G the function g, is upper semicontinuous on G x G. Note that, by (d), the function 
G>z + ge(a, z) need not be continuous (even for bounded domains of holomorphy). 


(g) For any G the function Ag is upper semicontinuous on G x C”. Note that, by (d), the function 
C">X + Ag(a; X) need not be continuous (even for bounded domains of holomorphy). 


Ste nis G2 nm Pu 
If vis @?, then (dd°u)” = det [Bester | a, at 


Example 7.4.4 (cf. [JP13], Proposition 7.1.3) Consider the Reinhardt domain G := {(z1, za) €C?: |z12z2| < 1}. 
Then 


m(a1@2,2122), ifa#0 
’ = ) € G, 
Bee oe ifa=0’ ~ 
y(a1a2)|a2X1 +a1X2|, ifa #0 5 
Ag(a;X) = , XX =(Xi,X2) eC. 
ala ) eens ra=0 ( 1 2) 


If b € Go := {(z1, 22) € G: z1z2 £0}, then ge(0,b) = |b1bs|'/? > |b1b2| = gc(b, 0). Thus gg is not 
symmetric. If D is a subdomain of G (with 0, b € D), then gp(0,b) > gp(b, 0) provided that D is sufficiently 
close to G. Thus there exist regular domains D C C? (e.g. bounded Reinhardt domains with real analytic 
boundary) such that gp is not symmetric. Observe that the invariant functions discussed before are all symmetric 


ones. 
If Goda, — 0, then 


gc(as, b) = M(a51452, bi b2) —- |b1b5| < |bybe|!/? = g(0, b). 


Consequently, gg is not continuous in the first variable. 
Let X = (X1,X2) € C?, X1X_ #0. Then 


A(as;X) = ¥(Gs,105,2)|@s,2X1 + as51-X2| >0< |X1X-|!/? = Ag(0;X). 
Hence, Ag is not continuous in the first variable. 


In the case where G is hyperconvex the pluricomplex Green function and the Azukawa metric are much more 
regular than in the general case (cf. [Dem87], [Kli85], [K1i91], [Zwo00b]). 


Theorem 7.4.5. (cf. [JP13], Proposition 4.2.10, Theorem 5.1.2). Assume that G is bounded hyperconvex. Then: 


li =1, G. 
(a) jim gc(a, z) ae 


(b) £ is continuous on G x G, where gc\¢xag := 1. 
(c) Ag is continuous. 
() Ag(a;X) = lim 77 Ba(a, a +AX), aeG,XeC". 
> 
Under some stronger assumptions gg and A are even in some sense Lipschitz (cf. [NPT09]). 


From a general point of view the invariant objects studied so far were defined via certain extremal problems 
related to: 


(a) holomorphic mappings f : G > D, e.g. me¢(a, z), ye(a; X); 
(b) analytic discs y : D > G, e.g. g(a, z), Kg (a; X); 
(c) logarithmically plurisubharmonic functions u : G — [0, 1), e.g. gg(a, z), Ac(a; X). 


At the end of the eighties E.A. Poletsky invented and partially developed a general holomorphic discs method, 
which in the meantime became one of the important tools of modern complex analysis. This method permits to 
reduce in some sense problems of type (c) to (b). It has various important applications, due to A. Edigarian (cf. 
[Edi02]) and E.A. Poletsky (cf. [Pol91], [Pol93], [EP97]). In particular, one may express the pluricomplex Green 
function in the language of holomorphic discs by the following Poletsky formula. 


Theorem 7.4.6 (cf. [PS89], [Pol91], [Pol93], [Edi97a]) 


gc(a,z)= inf Alordale-a) —: e(a,z), a,2€G, 
pe6(D,G) dEy1(a) 
p(0)=z 
aey(D) 


where ord, f stands for the order of zero of f at c. 


The idea of the proof (cf. [Edi97a]). The maximum principle for subharmonic functions implies that gg < ga. 
The system (&¢)g is holomorphically contractible. In particular, Eg < €%. Fix ana € G and let u := log (a, -). 
Observe that log gc(a, -) = sup{v € PSH(G) : v < u}. The main proof consists of two steps. 


(*) If p € @(D, G) is such that y(0) = z anda ¢ y(T), then 


Qn 
opeeG,2 <2 / u(o(et) at. 
0 


(**) The function 


U(w) = int{ 2 u(y(e"))dt : p € O(D,G), v(0) = wh, weG, 


is plurisubharmonic and @ = sup{v € PSH(G) : v < u}. 
Property (**) implies that %@ = log g¢(a, -). On the other hand, by (*), log &¢(a, z) < @, which finishes the 
proof. 


Exploiting the Poletsky formula, A. Edigarian proved the product property for the Green function (cf. 
[Edi97b], [Edi01]; some special cases had been solved before in [JP91c], [JP91d], and [JP95]). 

For any domains G; C C%, j = 1,2, we have gc,x@,((a1, a2), (21, 22)) = max{ge, (a1, 21), EG, (a2, Z2)}, 
aj,2; € Gj, j=1,2. Consequently, A has the product property 
Ag,xa,((a1, a2); (X1, X2)) = max{Ag,(a1;X1), Ac, (a2; X2)}, aye G;, X; E C'%,9 = 1,2. 


7.4.2 Sibony pseudometric 


Besides the Green function and the Azukawa pseudometric there are other invariant objects defined via extremal 
problems for plurisubharmonic functions. The most important one is the Sibony pseudometric, introduced by N. 
Sibony in [Sib81b] in the context of #-hyperbolicity. It is defined as 


Sc(a;X) := sup{ ((-2u)(a3 X))? :We \eusbSa(a) }, aceéG,xeEC", 
where 
\eusbSg(a) := {u: G > [0,1), logu € PSH(G), u(a) = 0, wis? near a}. 


Observe that the family (Sg)g is holomorphically contractible and yg < Sg < Ag. Moreover, S¢(a;-) is a 
seminorm for any a € G, and therefore, if G = G, is a non-convex pseudoconvex balanced domain (e.g. 
{(z1, 22) € D? : |z1z2| < 1/2}), then S(0;-) # Ac(0;:)=6 (see [Kli89]). On the other hand if 
G:={z€C:1/2 < |z| < 2}, then ye(a;1) < Se(a;1) = Ag(a;1), a € G (cf. [JP13]). In contrast to the 
previous invariant pseudometrics, the Sibony pseudometric is in general not upper semicontinuous (cf. [JP13]). 
Recent research was focused on boundary behavior of Sg, see e.g. [For09], [FLO9]. In particular, if 


G:= {zeC?:1/2< |lz| <1}, then 3 <Se¢(PjX) < + for sufficiently small t>0, where 
P, := (1/2 + 1,0), X := (1,0). Consequently, using [Kra92], we observe that 
ya(Pi; X) < Se(Pi; X) < kg(Pi; X), 0<t<1. Notice that the Sibony pseudometric is the “infinitesimal 


version” of the following holomorphically contractible family of functions 


sc(a, z) = sup{ Vul2) :We \eusbSa(a) }, a,zeG. 
Unfortunately, we have to confess that up to now almost nothing is known on properties of Sg. 


7.4.3 Bergman metric 


In the recent years the theory of the Bergman kernel and the Bergman metric became a very important tool for 
many questions in complex analysis (e.g. extension of biholomorphic mappings to the boundary). A detailed 
discussion of this topic is skipped in this survey by several reasons, mainly by lack of space. Nevertheless, for the 
convenience of the reader let us recall some basic definitions. 

For a domain G Cc C” let L?(G) be the Hilbert space of all holomorphic square integrable functions with the 


scalar product ( f, a) = i} fgd¥?". Then there exists a unique Bergman kernel Kg : G x G > C such that 
G 


. Kg(-,y) € Li(G), ye G, 

* f(y) = (f,Ke(.,y)), fi € L(G), ye G. 
Examples of Kg for different G can be found in the article by S.G. Krantz in this book. Recall that the Bergman 
kernel Kg of a domain G C C” isa @ ©-function on G x G. From now on we assume that Kg(z,z) > 0,z2€G 


(e.g. G bounded), which guarantees that the function z+ log Kg(z, z) is plurisubharmonic and @?. Then Kg 
leads to the following positive semidefinite Hermitian form 


n 2 


Boe 2, Oz, 0Z, 
vOZy 


vyp=1 


log Kg(z, z)X)X py 


and the Bergman pseudometric Bg (z; X) := »/Be(z;X),z € G, X € C”. Finally, set 


be(z', 2”) := (\smallintBg)(z’, 2”) = int{ [ Be (a(t); a'(t))dt : 


a: [0,1] > G isa piecewise @!-curve with a(0) = 2’, a(1) = 2}, 
z',2" EG; 


bg is the Bergman pseudodistance on G. We point out that the Bergman pseudodistance is invariant only under 
biholomorphic mappings, so from general point of view it is not holomorphically contractible. Nevertheless, we 
have yg < Bq for bounded domains G ([Hah76]). On the other hand there are bounded very regular domains 
G Cc C” such that Bg < Kg fails to hold ([DF80]). 

It has to be emphasized that the pluricomplex Green function turned out to be a very important tool in getting 
results of the Bergman theory. The reader who is interested in more details in the Bergman theory (for example, 
related to exhaustiveness and completeness) is invited to continue his study with diving into this new field, see e.g. 
[JP13], [Kral3], [Bto15], [Che15], [KZ15]. 
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8.1 Introduction 


Let's illustrate the above quote with a very simple example. 


(1) Consider a perfect “bell-shaped graph” f(x) = . If we take its 


Taylor series around the origin 


on ee 
1+x? 


f(x) = Ds (-1)"2, 


we note immediately that it diverges for all real x : |x| > 1. Why? Of 
course, the answer is clear, if we replace x by a complex variable z, the 
function f(z) := cs has two polar singularities at z = +i on the 
boundary of the circle of convergence. Thus, since the Taylor series 
naturally converge in disks in the complex plane, the presence of 
complex singularities interferes with the behavior of the series in the 
real domain. 


(ii) In the opposite direction, if we consider the Taylor series 


fiz) = 3 cos(./n )z” 


that converges only for |x| < 1, the function f(a) extends as a smooth, 
in fact a real-analytic function, to all real x < 1. In fact, if we consider 


ia) Ss" cos(1/n )z” as an analytic function of the complex 
0 


variable z, it extends as an analytic function to the whole complex 
plane C\{1} except one point z = 1, where f(z) has an essential 
singularity. How do we know that? 


There is an enormous amount of literature dedicated to studying the 
properties of analytic functions at large encoded in their local expansions in 
Taylor series—cf., the classical monographs by P. Dienes and L. Bieberbach 
((3], [4, Ch. X]). One of the first results in this direction is the following 
beautiful theorem of L. Kronecker ({4, Ch. X]). 


Theorem 8.1.1. (L. Kronecker, 1881 [12]) The Taylor series ae 


0 
represents a rational function f(z) = P(z)/Q(z), BQ are polynomials and 
max(deg P, deg Q) = N if and only if all the determinants 


[9072 


Q1°°*Qn+1 
det] . — Oforalln> WN. 


QAn*:*QAIn 


Our example (i) illustrates this theorem with N = 2. Since rational 
functions are obviously globally defined on the whole Riemann sphere, 
Kronecker's theorem is a very good example of a mandatory analytic 
continuation (single-valued as well) of a locally defined Taylor series. 

The example (11) is an illustration of a compilation of results of L. Leau 
—1899, S. Wigert—1900 and G. Faber—1903, cf. [4, p. 337 ff]. 


Theorem 8.1.2. (G. Faber, L. Leau, S. Wigert.) The Taylor series S- a,2” 


0 
with the radius of convergence I extends to C\{1} if and only if there 
exists a (unique) entire function g(z) of order zero (minimal type) such that 
An = g(n), n= 1,2,---. If g(z) is a polynomial of degree m then 1 is a 
pole of orderm + 1. 


(Recall that an entire function g is called of minimal type if for every 
€ > 0 there exists a constant C, such that | f(2)| < C-e8!,) 


1 
1—z 


(ee) 
For example, the geometric series Ss" = illustrates the latter 
0 


part of the theorem with m = 0. 

To illustrate the ideas behind this and similar results, let's sketch a 
slightly more modern result of this type due to T. Qian ([{14]) and D. 
Khavinson [8] (the latter with a different, much shorter proof). 


Theorem 8.1.3. [8, 14]. Let f(z) = »,, b,z” be an analytic function in 
1 


D := {|z| < 1}, and b,, = g(n), where g is of minimal type in the sector 
Sp := {z: |argz|<y,0<y< 4}. Then, f(z) extends to the “heart- 
shaped” domain 0, := {z = re’, 2n —coty-logr > 6 > cot y- log ae 


Note that when y = F, Q, = C\[1, +00). 


Sketch of the proof [8] following ideas of Le Roy and Lindel6f [4, p. 
340 ff].] By the residue theorem 


: Sedge e2miw _ 1’ 
(8.1) 


where y is any contour in S, enclosing the integers 1,---, NV and no others. 
(Can choose, eg. for y the boundary’ of the — sector 
{w: | arg(w — a)| < y, |w— a <R,O0<a< 5, R=N+3}) 
Y=%7~U eR, where yp is a circular part of the boundary, while y, 
comprises two sides of the angle with vertex at a. 

An elementary argument yields that for all 
weCc\ 1h {z:|z—n|<7,n > 0-small}, jee" —1]>ce>0, 
c = c(n) is a constant ([4, p. 341]). Hence, the part of the integral (8.1) 
restricted to yp tends to zero when R — oo (i.e., N — oo) for z= —r, 
0<r<_1,i.e., zin D and on the negative radius. (This is seen from an 
elementary estimate |e” — e~™”| > ce™*lsin§] © for w=at Re”, 
assuming at first ¢ < 2.) Thus, for z € (—1, 0] in D, 


(8.2) 


where I), := jim 15 = fu -arg(w —a) = to} Thus, in order for the 
— 00 


integrand in (8.2) to decay exponentially on I, we obtain using the 
assumptions on g, that it suffices to have 


cot y - log |z| < arg z, for arg(w — a) = y 
and 
2x — cot yp: log |z| > arg zfor arg(w — a) = —y. 


This proves the statement for y < oe since (8.2), an analytic function in z, 
converges for all z€ , and coincides with f on (—1,0). Finally, 
Q2 a= U_ ‘ee LI 
ens 

Remark. The assumptions on g can be relaxed somewhat further (to g being 
of exponential type less than m—cf. [4, pp. 341-342]). It is not known 
whether {2,, is the largest domain one can extend f(z) to. Thus, the “only 
if’ part is still missing, unlike for classical results of Kronecker and Leau 
(Thms. 8.1.1 and 8.1.2). 


Of course, it is impossible in an article like this one to survey all 
beautiful topics investigated in the classical avenue of analytic continuation: 
monodromy, continuation of algebraic functions, over-convergence and gap 
series, universal Taylor series, and many others. We hope that an interested 
reader will be tempted to continue research on her own: [4, Ch. X, XI], [3] 
are good books to start. The more recent vast literature on universal Taylor 
series can be found on MathSciNet. 

From these classical themes of continuation of Taylor series, let's make 
a leap to the problem of analytic continuation of solutions of most basic 
equations of mathematical physics. 
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8.2 Continuation of Solutions of Linear PDE 


8.2.1 ODE vs. PDE 


It is well known that the solution of the initial value problem for the linear 
ODE 


w)(z) + ana(z)wO(z) + +++ + an(z)w'(z) + ao(z)w(z) = F(z), 


w(0) = wo,--- ,w' (0) |) ea 
(8.3) 


with the coefficients as and fanalytic in a domain © containing the origin, 
extends as analytic function throughout 0. (It might end up being multi- 
valued, if © is not simply connected, i.e., has holes, but nevertheless can be 
analytically continued everywhere in 0.) 

Yet, if we consider a very simple initial value problem for PDE: 


ie = g? Se w(x,0) = 2; 


(8.4) 


1—z 
line {y = 0} on the hyperbola y = 1 /x. How can we explain this? 

Moreover, if we consider a more general initial value problem for (8.4) 
with arbitrary “data” w(x,0) = f(x), where fis a polynomial, or an entire 
function, one easily checks that the solution is 


the easily found solution w := blows up arbitrarily close to the initial 


w(x, y) = rk ca J) 


This is striking since it yields that the variety I := {ay = 1} is the only 
possible carrier of singularities for all solutions to (8.4), independently of 
the data, as long as the data itself has no singularities. 

Let's postpone the heuristic explanation of this fact till later and discuss 
another natural problem of analytic continuation coming from mathematical 
physics. 


8.2.2 G. Herglotz’ memoir of 1914 


The following question was first tackled by G. Herglotz in 1914 (also, cf. 


[15, 18]). Imagine a solid in R°, or a “plate” (a domain) in R?, 0, bounded 
by, say, a nice algebraic surface (or, a curve). 
Let 


uo(x) = A kn(a, y) dy, 
(8.5) 
where 


kn (x,y) = 3, log ea n=2 


a ee _ 
An rar 


be the “gravitational” or “electrostatic” potential of 0. Obviously, u is 
harmonic outside ©, and the natural question tackled by Herglotz was: how 
far can one harmonically continue u(x) inside QO before running into a 
singular point? (Herglotz did answer the question in R? and made some 
headway in R°, but because his prize-winning memoir appeared on the 
brink of World War I it fell into oblivion while most of the results were 
rediscovered by other authors—cf. the references in [10, 16]. For example, 
if Q is a disk or a ball, then, of course, (e.g., n = 3) by the mean value 
theorem, we have: 


1 dy __ const 


uo(z) = —-Z io 
{yl<a} |e — y| 


(dy, of course, stands for the Lebesgue measure in R”.) 


So, ue(«) extends as a harmonic function everywhere in R? except for 
the center of the ball. This goes back to I. Newton and is well-known. What 
is perhaps less well-known is that conclusion stays true for 


went [2M o-foc 


In 


with an ARBITRARY polynomial, or even an entire density p(y). In the 
latter case, all the symmetry associated with the ball goes out the window, 


and the conclusion that ug, extends harmonically to all of R°\{0}, 
matches Leau's theorem from Section 8.1 in mystery and beauty. 

Herglotz’ problem is often restated in more “physical” terms: consider 
the exterior gravitational potential of an analytic mass density po(y) in the 
region (. Find a smaller object E inside 0 and a different mass-density p, 


on £ that is gravi-equivalent to po, 1.e., such that the potential uz,, and 
UQ,p, coincide outside of 0. 


Example 8.2.1. For 2Q= oi || < 1} Cc R® (or, more generally, R") 
Po = polynomial of degree < N, E = {0} and p, is the distribution of 
order < JW at the origin. 


Example 8.2.2 (cf., [9, 10, 16]). An oblate spheroid 
x? x2 x 
= {eeR': 34+ 343 <1a>b>0} 


(planet Earth, e.g.). Then, for say, uniform density po = 1 (or any other 
polynomial, or, entire density) u(x) extends into 2\ E, where 


Es {x3 = 0,2? + x2 < a® — b*} 


is the caustic disc. The relevant density p,; on E (relevant to po = 1) is 


algebraic and equals const (a? b? 24 3) M2 of [10, Ch. 15]. 


2 
Example 8.2.3. A prolate spheroid (2 := {Z + 4 + 3 <1,a>b> o} 
gives a completely different picture, an exciting mystery on its own. The 


potential wg extends to 2\E, but Fin this case is a 1-dimensional segment 
{2 = 23 = 0,|21| < Va? — oS, while the density py = (a? — b? — 23) 


is a polynomial. Moreover, below we shall touch upon the rather deep 
problem regarding the dramatic differences in singularities of we in the 


latter two examples: bounded, a square-root type singularity in the former, 
and unbounded—in the latter. 


8.2.3 A further discussion of the Herglotz question 


As one readily obtains directly from (8.5) via Green's theorem, 


Aun = xa; 
(8.6) 


1, «Ef 
where x e(x) = i __ and stands for the characteristic function 
0, «zER\N 
of QO. Denote by M, the so-called modified Schwarz potential (of 092) the 
solution of the following initial value problem 


AM = 1 near I’ := OO”; 
M=VM=O0Oonl. 


(8.7) 


(The solution exists and is unique by the Cauchy—Kovalevskaya theorem— 
cf. [10], e.g.) Then, the function 


ee cr outside {2 
“~ lug—M = inside 2 


(8.8) 


gives the desired continuation. Indeed, ug — M is harmonic in 0 near [ 
and coincides with uy on I together with its first derivatives. The statement 
then follows by a straightforward application of Green's formula—cf. [10, 
Thm. 6.1]. 

For an arbitrary polynomial or entire mass density p, we only need to 
modify (8.7) and define M, as a solution of the initial value problem 


AM, = pnear I’; 
M, = VM, =0onl. 


(8.9) 
If instead of M we consider the Schwarz potential ur of I defined by 
Aur = 0 near I’; 
‘ = = Fale onl’; gradur = 1 Zon r 
(8.10) 


(n = 2 or 3, as in our examples), then obviously, Mp = = Fal — up. 
Similarly, M, = Q —ury, where Q is a polynomial, an entire function 
such that AQ = p, and ur accordingly defined as a solution of the initial 
value problem similar to (8.10): 


Aur» = 0 near I’; 
Urp = Q, Vuryp = VQ onl. 


(8.11) 


Thus, if we could show that the singularities of any initial value problem for 
the Laplace operator posed on I are only dictated by I itself, not by initial 
data (+ |x 2 or Q), we would have achieved the high ground needed for 
understanding the Herglotz’ problem. A deep and beautiful theory of Leray 
explains the origins for the appearance of singularities of initial value 
problems near initial surfaces—cf. [10, Ch. 13, 19-20] and Leray's original 
papers referenced there. 

Indeed, generically, it asserts that the singularities appear and take off 
(locally, sic!), from the initial surfaces at the same places and along the 
same routs independently of data. Moreover, in dimension 2, and also, in 
higher dimensions, but only for quadratic surfaces, it has been proved that 
the local theory of Leray, first verified only near initial surfaces, holds 
globally—cf. [10]. Here, we will simply illustrate the Leray principle by a 


couple of straightforward examples. We only sketch the main steps, more 
details can be found in [10]. 


Example 8.2.4. Let Q= {or =f e —1<0,a> b} be an_ ellipse, 
P= 087: 
One can calculate wr, and then further 


1 — Our __ 
3 Vur = az 


= z 


21 p72 _ = 
ath a a Ce, ce _ a’ — b? 


(8.12) 


(z= 2x + 1y, as usual). So, the singularities of wr are at the foci of 0. The 
solution of the initial value problem (8.10) is fine by the Cauchy-— 


Kovalevskaya theorem near complexified quadratic curve Tin 2, 


— 


2 2 
P= {2yec:4+8-1=0} 


except for 4 points {(+4,4+4)} on [° where C-K theorem breaks 


down. From those “bad” points, as Leray's theory asserts, the singularities 
travel along 4 complex (characteristic) lines 


{ (x,y) EC? :atiy= const } tangent to I at the above characteristic 


points. These lines, the “carriers” of singularities, reach the “real” space R? 
at the foci (+c, 0) of the ellipse. Since the carries of singularities depend on 
I only, logarithmic potentials of Q with arbitrary polynomial, or entire 
densities exhibit the same behavior and might become singular only at the 
foci as well. 

Moreover, when 6 t a, c | 0, an ellipse becomes a circle, the “bad” 


points on T all move to infinity, and the singularities of (8.12) change from 
algebraic (,/-type) to polar-< at the origin, the limiting position of the 
collapsing foci. 

A similar but technically much more demanding analysis provides the 
justification for Examples 8.2.1-8.2.3 in R°, and in general in R’—cf. [10] 
and the works of G. Johnsson referenced therein. However, we emphasize 


that for algebraic surfaces of degree > 3 in R", n > 3, the analysis of 
Herglotz’ problem, 1.e., the global version of Leray's principle, is still 
waiting to be discovered. 


We shall finish this section with another transparent example illustrating 
Leray's theory. 


Example 8.2.5. Consider the initial value problem 


Oru a sin Syels 
Qaby = 9, near := {y= 2°}; 


fusy, Se = xonl. 


(8.13) 


One readily finds the solution u(z,y) = = ee ue that is “ramified” 
around {y = 0}. The latter is, in fact, Leray's characteristic tangent to I at 
the (unique, w.r.t. iy operator) “bad” characteristic point (0, 0). 

In higher sien: the situation is more complicated. In a nutshell, 
there are more “complex characteristic lines” tangent to the initial surface 
that we view as continuation of I into C”. These lines carry out singularities 
off the initial surface. The analytic functions having singularities on a piece 
of an analytic hypersurface however must be singular on the whole 
hypersurface by a celebrated theorem of Hartogs (in C”, n > 1, of course). 
In other words, the singularities propagate from “bad” points on the 
complexified (embedded into €”) surface I’ and then exhibit themselves in 
KR" at points where the Leray characteristic tangent, the carrier of 
singularities, hit KR”. This is transparent and proved rigorously (cf. [10, Ch. 
13], e.g.), by G. Johnsson for quadratic surfaces in K"—cf. [10, Ch. 19-20] 
and references to Johnsson's original papers contained therein. This also 
explains the difference in the nature of singularities in Examples (8.2)— 
(8.3). In the of the oblate spheroid, each point on the circular caustic 
{ee + 2% <a? —b?,23 = = 0} is a “meeting point” of true characteristic 
lines coming from C and tangent at characteristic points on the 


2 2 2 
complexified surface { (1, #2, 23) aCe + + + 1=0,a> b} 


For the prolate spheroid, each point of the caustic segment 
{lal < Va? — b?, 2. = 23 = o} is a meeting point of infinitely many 
characteristics, thus causing unbounded singularities. So, intuitively, the 
idea that “more carriers of singularities meeting at a point in R”’ should 
result in a “heavier” singular behavior is tempting and reasonable. However, 


essentially, nothing has been rigorously proved along these lines. A worthy 
and challenging avenue for further research. 
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8.3 Analytic Continuation and Problems of Uniqueness 


Consider the spherical shell 2:= 12 ER?:r< |x| < R} ( 
jz|? = 2} +25 +23, as usual). Let u be a harmonic function in Q that 
vanishes on the segment (—R,—r) of the x,-axis. Then, let us pose the 
question (cf. [10, Ch. 9]). 


Question 8.3.1. Must u also vanish on the segment r< 2, < R, 
2 = %3 = 0? 


The same question in 2 dimensions can be easily settled by elementary 
complex analysis. To fix the ideas, let r= 1, R = 2, n = 2. Consider the 
harmonic function v(z) := u(z) + u(Z) in the annulus Q := {1 < |z| < 2} 
. By the Schwarz effection principle, v = 0 in a small disk centered on 
(—2,—1), say, ee jz+ 3] < +}. Hence, v = 0 in Q and, accordingly, 
2u(x) = v(x) = 0 on (1,2). Thus, in this situation, the answer is “yes”, 
Although, the answer is also “yes” in R", n > 3, the above argument of 
course, fails. Moreover, the above argument doesn’t work either if instead 
of a line through the center of the annulus, or a spherical shell, we consider 
an arbitrary line still cutting © in two disjoint segments. It might come as a 
surprise that the answer remains “yes” if J > 3 (a thick annulus, or shell), 
but becomes “no, not necessarily” if ft <3 (a thin annulus) and the 


constant 3 is sharp (cf. [10, Ch. 9], [11]). Even more intriguing ({10, Ch. 


9]), the same question posed for a torus in R?, has always a negative answer 
in general. 

What is the high ground for this question? The answer is analytic 
continuation. Indeed, a harmonic in a domain (2 C R", n > 2, function u 
automatically extends as a holomorphic function of n variables to a domain 


Q in C. Q can be viewed in a rather simple way as follows. For all 


2° e R"\Q, consider the isotropic cone 
Daa ee CU: z,-«°)* =0 Then, 2:— C” UU Dgo 

PECL Ga =9) ki 
cf. [1, Ch. 1]. 


The beautiful fact established in the theory of linear analytic PDE is that 
solutions of ALL PDEs (A”+ lower terms) u = f with, say polynomial or 
entire coefficients in ( automatically extend to Q. For example, if 
= {|x| < 1} is the unit —_ ball in R®, i 2, 

s 1/2 
2 
» 


2 
Q=<zEC": Ci — + ||z||? <1 >, the celebrated Lie 
i 


ball (cf. [1, 2, 10]. (For n=2, Q={(X,Y) €C?:|X+éY|}, the 
bidisk (cf. [1, 2, 10]). Thus, a sufficient condition that would yield an 
affirmative answer to our question, is whether the intersection 
Qn {Y = o} the harmonicity hull of the shell O and the complexified line 
{y = c} (in dimension 2, e.g.) is connected or not. In the original question 
for the annulus, for example, this intersection becomes 
{(X,c):r< |X +ic| < R} and is disconnected if 4% <c<r, and 
connected if 0<c< a. If a So) Oa. a >r>c, so the 
intersection of and the complex line {Y = c} is connected. The fact that 
the constant 3 is sharp is seen (T. Ransford) by taking Q to be an annulus 


separating {0,—i} from i, and u(z) := Re W/z(z— i)(z +i), where we 
can take any branch of the square root. Then, u(x) =0, «<0 and 
u(x) > 0, « > 0. ft < 3 but can be made arbitrary close to 3—see [11], 
[10, Ch. 9] for more details. 


As is remarked in [11], this simple consideration allows to answer 
questions similar to Q. 8.3.1 not only for solutions of linear PDE with the 
power of the Laplacian in the principle part but also for functions 
represented by arbitrary Riesz potentials, the latter, in general, need not 
satisfy any linear PDE. 


Dr 


8.4 Analytic Continuation of Series of Zonal Harmonics and 
Series of Orthogonal Polynomials 


By analogy with analytic continuation of Taylor series, let's consider the 
problem of finding singularities of other series expansions. To fix the ideas, 
let 


t= SS anr” P,,(cos 8) 
n=0 
(8.14) 


be an axially symmetric harmonic function in the unit ball in R°. a, € R, 


Jim |a,,|"/” =1,P,(z) = oor - | (2? — 1)"] are Legendre polynomials 
(orthogonal on [—1, 1], || Pa lls = seq): — (e? + 22 + 22)" die 


distance to the origin, 0 is the usual azimuth angle in spherical coordinates. 
One can easily verify that the expansion (8.14) diverges for r > 1, so u 
must have singularities on the unit sphere S? := {r = i The question is 
where? The following remarkable theorem was proved by G. Szego in 1954 
[17]. 


Theorem 8.4.1. u(r,@) extends harmonically across the circle 
(1,09 :0 < y < 27) on the sphere S 2 if and only if the Taylor series (!) 


CO 


f() := S- Ané” extends across £9 := e*. 


k=0 


This is a truly amazing result, since at first glance, the expansions in 
zonal harmonics and the Taylor series built with the same coefficients 
should have nothing in common. Moreover, inspired by Szego's theorem, 
Nehari proved the following beautiful follow-up [13]. 


1/n 


Theorem 8.4.2. Let {a,,} € C, satisfy lim la,|'/" = =, R>1 and let 


ii) = SS AnP,,(t) be (as is easily checked) an analytic function inside the 
n=0 


ellipse Dr with foci at +1 and sum of whose semiaxes equals R. In other 
words, Dr := { (2,9) ‘ + x <la+b=R,a-—b= +}. P(t), as 


in Szegé's theorem, denote Legendre polynomials. Then, f(t) is analytically 
continuable across t,€ OBR if and only if the analytic function 


(ee) 
g(s) = S- ans” 
n=0 


s=y(t) =t+VeP—-1, ge: €\[-1,1] > {seC: |s| > 1}, 
(oo) = 00, is analytically continuable across the corresponding point 
0 = (to), (to = + (80+ 89°), 80 € {8 : |s| = R}. 


s| < R, where s and t are related by the conformal map 


Once again, the reader should observe how Nehari's result, 
unexpectedly, connects the singularities of the expansion in orthogonal 
polynomials with seemingly disjoint Taylor series. 

Since there is a wide variety of results allowing one to identify 
singularities of Taylor series on the circle of convergence, the above two 
results provide a powerful tool for identifying singularities of harmonic 
functions and orthogonal polynomial expansions. 

Both theorems can be significantly extended by replacing Legendre 
polynomials with arbitrary Jacobi orthogonal polynomials (R. P. Gilbert 
(1969), P. Ebenfelt-D. Khavinson-H. S. Shapiro (1996))—cf. [10, Ch. 10], 
[5], [6] and references therein. 

Recall that for a,@ > —-—1, the Jacobi polynomials are orthogonal 
polynomials on [—1,1] with respect to the weight (1—x)°(1+2)’, 


normalized by P“"(1) = (" ’ *), So, a= 8=0 corresponds to 
n 


Legendre polynomials, a = 6 = -) —Tschebysheff polynomials (that, in 
turn, under a suitable change of variables, correspond to monomials z’”, 
n > 0 in the unit disk—cf. [5, 6, 10]; a= B= ‘3 k > 2 being an integer 
corresponds to ultraspherical polynomials appearing in expansions of 
axially symmetric harmonic functions in R* (Gegenbauer polynomials). 

Now, the original proofs of Thms. 8.4.1 and 8.4.2 boil down to writing 
down the given expansions in terms of a certain integral and ingenious 
manipulation of the latter. The “high ground” approach advocated and 
developed in [5, 6] consists of noticing that ALL relevant expansions in 
general Jacobi polynomials can be interpreted as solutions of a Cauchy 
problem for a linear PDE in two variables with the same initial data. Of 
course, the partial differential operator corresponding to every particular 
expansion is different in each case. But all of them share the same principal 
part, the part of the differential operator that involves the senior derivatives. 
A deep result in the theory of linear PDE, based on the 1970 extension by 
M. Zerner (cf. [10, Ch. 4] of the classical Cauchy—Kovalevskaya theorem 
yields that the singularities of the solutions of the Cauchy problems locally 
depend exclusively on the principal part of the differential operator. Hence, 
all the expansions in Thms. 8.4.1 and 8.4.2 share the same singularities thus 
unveiling the mystery behind Szeg6's and Nehari's results. 


8.5 An Epilogue 


This article's only intent is to initiate for the curious reader a few possible 
modern directions in the classical theme of analytic continuation. There is 
absolutely no way to cover all possible topics, thus our choices were limited 
to several topics the author felt most comfortable with. 

There are so many themes that were left out entirely, e.g., beautiful 
results of Eisenstein regarding algebraic properties of Taylor series 
depending on properties of coefficients—[3, 4]. Essentially, no deep and 
subtle results, starting from Painlevé classical researchers, dealing with 
classification of singularities and analytic continuation of solutions to 
nonlinear ODE in the complex domain—cf., e.g., [7]. In classical potential 
theory, we left out beautiful modern generalizations due to A. Givental of 
the Newton's “no gravity in the ellipsoidal cavity” theorem, Ivory's theorem, 
MacLauren's mean value theorem for ellipsoids, viewed from the modern 
viewpoint of analytic continuation of Cauchy's problem for the Laplace 
equation—cf. [9, 10] and references therein. The theme of analytic 
continuation of solutions to the Dirichlet problem in domains with algebraic 
boundaries is far from developed and has an attractive array of important 
open problems, even in two dimensions—cf. [10, Ch. 18]. Even more basic 
open problems await an interested reader if one extends the search for 
singularities of solutions to the classical Dirichlet problem for the Laplacian 
to that for Helmholtz’ equation. In other words, expanding the program to 
study possible singularities of the eigenfunctions of the Laplacian in 
domains with algebraic boundaries. The reigning open conjecture that 
ellipsoids are the ONLY domains for which al/ eigenfunctions are entire 
(and of exponential type) remains virtually untouched. 

A fairly recent solution by P. Ebenfelt and D. Khavinson—cf. [10, Chs. 
11, 12] of the problem of reflection of harmonic functions across analytic 
hypersurfaces in higher dimensions (or, why doesn’t Schwarz reflection 


principle work in, say, R>?) opens up a new venue for investigations: the 
“antenna problem”. In short, it is the question of the possibility of reflection 
from a point to a compact set vs. point-to-point reflection. 

Once again, more important for applications is the reflection question 
for solutions of the Helmholtz equation, i.e., the eigenfunctions for the 


Laplacian. That playing field is widely open as well—cf. [10] and 
references therein. 

Finally, we have only mentioned in passing the powerful methods of 
analytic continuations of solutions of linear analytic PDE combined with 
the modern techniques of several complex variables. The results culminate 


in Leray's theory of propagation of irregularities through C”. The underlying 
techniques based on the so-called method of “globalizing families” is both 
clear and quite powerful—cf. [10, Chs. 4-10, 19, 20] and references 
therein. 

However, at present, the theory is more or less complete (we mean the 
global theory of propagation of singularities) mostly in two variables and 
also in n > 3-variables, but there exclusively for singularities initiated on 
quadratic surfaces [10, Ch. 19-20]. Once again, the importance of the 
remaining open problems is difficult to overestimate. 

In conclusion, by this short survey, we wanted to demonstrate that the 
classical theme of analytic continuation of functions of one variable that has 
intrigued researchers for at least 200 years since the concept of an analytic 
function had come into focus, is alive, doing well and is quite rich with 
plenty of attractive and beautiful problems, conjectures, and attractive 
routes for further study. Thus, we hope that this small survey and the 
appended references will prompt the reader to invest time and effort in 
further research on these truly “eternal” topics. 
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9.1 Introduction to This Chapter 


This chapter is devoted to complex convexity. Which are the most significant results presented 
here? This is the question that I answer in this first section of the chapter. But first I shall explain 
why real convexity is of interest, why complex convexity is important, and why mathematical 
morphology is a useful tool in the study of convexity. 

What makes the approach in the present chapter different from other presentations of the 
subject? Also, this question will receive an answer. 


9.1.1 Why is real convexity of interest? 


A subset A of R” is defined to be convex if for any pair {a, b} of points in A the whole interval 
[a,b] is also contained in A. A function f : R” — [—oo, +00] is said to be convex if its finite 
epigraph 


epifmte(f) = {(2,t) € R" x R; f(x)<t}, 
(9.1) 


is a convex set. 


Convex functions possess a property of great importance in optimization theory: A local 
minimum of a convex function R” — R is automatically a global minimum. In other words, if 
f :R” => R is convex and f(x)>c for all points x in a neighborhood of a, however small, then 
f(x)=c for all x in R”. 

Real convexity appears naturally also in complex analysis: the indicator function of the 
Fourier transform (defined in C”) of a function or distribution in R” of compact support is 
convex. 

We collect in Section 9.2 definitions and basic properties of convex sets and functions in 
vector spaces over the field of real numbers or over the field of complex numbers. 


9.1.2 Why is complex convexity important? 


In one complex variable, complex convexity of sets is not of great importance. Given any open 
set Q of the complex plane C and a point p not belonging to 0, we define a rational function 
z+ 1/(z—>p) which cannot be extended as a holomorphic function across p. 

But in two variables, the set of singularities of a rational function, indeed of any 
meromorphic function, cannot be just a singleton set. There are easy examples of two sets w C 2 
such that any holomorphic function in @ can be extended to a holomorphic function in 0. 

This phenomenon gives rise to the concept of domains of holomorphy, which are domains 
such that there are holomorphic functions that cannot be continued to a larger domain, in a sense 
to be made precise. Related to these is the definition of a pseudoconvex domain. That a domain 
of holomorphy is pseudoconvex was proved by Eugenio Elia Levi (1883-1917); the converse, at 
the time an unsolved problem, came to be known as the Levi problem. It was solved by Kiyoshi 
Oka (1901-1978) in two variables, and later in any finite dimension by Oka, Francois Norguet 
(1929-2010) and Hans-Joachim Bremermann (1926—1996)—for a survey, see (Slatyer 2016). 

So these phenomena point to the fact that there are great differences between the geometry of 
C and the geometry of C?. We can draw two-dimensional figures on a paper, and we can visualize 
objects in three dimensions. Nowadays there are even nice programs that create figures on the 
screen that can be rotated to exhibit all properties of an object in three-space. But two complex 
variables is a challenge because they correspond to four real coordinates. 

Can you see in four dimensions? Yes, it is indeed possible to train one's inner eyes to see in 
four dimensions. A nontrivial but most rewarding sport. We can actually arrive at true 
stereoscopic vision ... However, if you are not yet a master of four-dimensional landscapes, you 
will appreciate the Hartogs sets, named for Friedrich Moritz Hartogs (1874-1943), where we can 
be content with three real variables (Rez1,Imzj,|z2|) instead of the four 
(Re z1, Im z1, Re z2, Im zz). An example is Figure 9.1 on page 281. To view Reinhardt domains, 
named for Karl Reinhardt (1895-1941), we need only (\z1 F |Z2 ») € R?. 


9.1.3 Why is mathematical morphology a useful tool in the study of convexity? 


Mathematical morphology can be superficially described as applied lattice theory. As such it is 
about the operations (x, y) +> x \ y = min(z, y) and (z,y) + xz V y = max(z, y) in an ordered 
set. These operations replace addition and multiplication in a ring, and an important example is 


the Boolean ring of all subsets of a given set, with A as intersection and V as union. 
In convexity theory, we see that the intersection of two convex sets is convex, while the union 
is, in general, not. But we still have a lattice, in that the convex hull of the union of two convex 


sets is the smallest convex set containing the two, and therefore is the supremum of the two. This 
can then be done analogously for convex functions, and, more generally for plurisubharmonic 
functions. It turns out that complete lattices are important; they are the ordered sets which allow 
infima and suprema also of infinite families. 

Mathematical morphology provides us with important concepts in the theory of ordered sets 
that are helpful in understanding several related phenomena in mathematics. See Section 9.3 for 
more details. 


9.1.4 Which are the most significant results reported in the present chapter? 


An important observation is the non-local character of lineal convexity for general sets. As 
always, properties such that the local and global variants are different create difficulties—which 
may be challenging. 

Because of the non-local character just mentioned, it is of importance to know that for 
bounded sets with a smooth boundary, the property of being locally lineally convex actually 
implies the global property. This is proved in Section 9.6. 


9.1.5 What makes the approach in the present chapter different from other presentations 
of the subject? 


Complex convexity is quite a well-studied field, and the ways to approach it are not many. 
However, we shall view convexity from the inside as well as from the outside, and this gives 
perhaps interesting perspectives. A set is concave if and only if its complement is convex, and the 
two notions should be studied together. 

As mentioned, a subset A of R” is defined to be convex if for any pair {a, b} of points in A, 
the whole segment /a,b/ is also contained in A. This is what we can call convexity from the 
inside, 1.e., looking at subsets of the given set. But we can also look at the set from the outside: If 
P does not belong to A and A is open or closed, then there is a half-space that contains A but not 
p. This is the Hahn—Banach theorem, of utmost important in convexity theory, both in finite 
dimension and infinite dimension. Explicitly, we say that a set in a vector space over R or C is 
lineally concave if it is a union of hyperplanes, and lineally convex if its complement is lineally 
concave. In one dimension, hyperplanes are just points, so every set is both lineally concave and 
lineally convex. In higher dimensions a convex set need not be lineally convex, but if it is open or 
closed, this is true. All this is true both in the real and the complex settings. For more details, see 
Section 9.4. 
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9.2 Introduction to Convexity 


9.2.1 Introduction to this section 


The theory of convexity of sets and functions in vector spaces is a highly developed and very rich 
theory. We collect in this section definitions and basic properties of convex sets and functions in 
vector spaces over the field of real numbers or over the field of complex numbers. However, we 
present but a bare minimum of what is needed for the rest of this chapter. For fuller accounts, see 
the classical book by R. Tyrrell Rockafeller (1970, 1997), the books by Jean-Baptiste Hiriart- 
Urruty and Claude Maréchal (1993, 2002), and also my book manuscript (ms 2021). 


9.2.2 Sets, mappings, and order relations 


9.2.2.1 Notation for numbers, norms, and derivatives 


We write N = {0,1,2,...} for the set of natural numbers, following Bourbaki (1963:67),! and 
Z, R, C for the ring of integers, the fields of real and complex numbers. 
1/p 


We shall use the /-norm lzlp =O P) , 1<p<-+oo, and the 1/°-norm 


J 


25 


I|Z||oo = sup | for z € C”. When any norm can serve, we write only ||z]]. 
j 


The bilinear inner product of two vectors in R™ or C” shall be denoted by a dot: 


EYH UY te + lmYms 2° WS 2Wi +--+ + ZnWn, z,y€R”™,z,w eC”. 


The Euclidean norm will be written like this: 


Itl2=Vz-a; — |[z2=Vz-z, 2eER™, 2 EC". 


We shall denote by Bz(c,r) and B<(c,r) the open ball and the closed ball, respectively, 
with center at c € C” and radius r € R for any norm, thus 


Be(c,r) = {z € C*; ||z-—el| < r} and B<(c,r) = {z € C’; ||z — el|<r}. 


If n = 1, we shall write instead D<(c,r) and D<(c,r) for the disks. 
The closure, interior and boundary of a subset A of a topological space will be denoted by A, 


AU and OA, respectively. Thus Bz(c,r) = B<(c,r) if ris positive, and B<(c,r) = Be(c,r) 
for all real r. 
For derivatives of functions we shall use the notation 


_ of _ Of —_ Of _ 1 : 
fa; = Ox;? fy; = Oy;? t= Oz; x (So; ify,), 
1Alfred Tarski (1956:121) calls 0 a natural number. 


of 1 : 0? f . 
fe, = Ge = 7 (fe; + ify), faite = Faom? j,k=1,...,n. 


Differentials are written as 
df =d'f +d’f, whered’f=§°f,,dz;andd’f= Sf, day. 


(9.2) 


9.2.2.2 Counting with infinities 
We shall also use a notation for the extended real line 


R; = RU {—00, +00} = [—00, +00], 


thus creating the two-point compactification of R by adding two infinities, +oo and 
—oo = —(+00). We also add these infinities to the integers, 


Z; = ZU {—co, +00} = [—00, +00]. 


Similarly we write Y; = Y U {—o0, +00} for any subset Y of R. 

How shall we define a sum like (+00)+(-—oo) ? Can addition 
R x R3(z,y) > «+ y€ R be extended to an operation R; x Ri3(z,y) 9 ex+yeER, ina 
reasonable way? A convenient solution, pioneered by Jean-Jacques Moreau (1923-2014) in his 
paper (1970), is to define two extensions, upper addition and lower addition . The first is an upper 
semicontinuous mapping from R; x R: into R,; the second a lower semicontinuous mapping. 


They are denoted by + and + and are defined by the requirements of being commutative and to 


satisfy 


z+(+oo) =+00forallx € Ry; 


az +(—oo) =—ooforallx € [-o0, +00]; and 
oy = —((—z) + (—y)) forallx, y € Ru. 
(9.3) 
When there are several terms we may use the summation symbol with a dot: 
Ls . . 
Softy =tit-+tm, te Rs. 
j=l 
(9.4) 


A convenient rule is the following. 


Lemma 9.2.1. For any element c € Ry and any function f : X — Ry, defined on an arbitrary set 
X we have 


inf (c+ f(x) =¢ + inf f(a). 
(9.5) 


Note that there are no exceptions to this formula. 


Proof We just need to check all possibilities where our intuition is less reliable than usual, i-e., 
when c = too or X is empty. 


We also note the equivalence 
a+b>c 4 a>c + (-), a,b,c ER». 
(9.6) 


9.2.2.3 Sets 


The empty set, the unique set with no elements, will be denoted by ©. If A and B are sets such 
that every element of A belongs also to B, then A is said to be a subset of B and B a superset of 
A. This is written A C Band BD A. 

The family of all subsets of a set Wis called the power set of W, and will denoted by A(W). 
Thus A € A(W) if and only if A C W. We denote by Prinite(W) the family of all finite subsets 
of W. 

We shall use the usual symbols for the intersection and union of a family (A;) jeg OF sets: 


al A; and U A;, A; € PW). 
jeJ jeJ 


(9.7) 


When the index set Jhas only two elements, we write A; M Ag and Aj U Ag. 
The set of all elements in A that are not elements of B is called the set-theoretical difference 


of A and B, written A\B. When A is equal to the whole set W we can write CB for W\B, the 
complement of a subset B of W. 

The Cartesian product of two sets X and Y is the set of all pairs (x, y) with x © X andyc Y 
. The Cartesian product of n sets X;, 7 = 1,...,n, denoted by 


Xix---x Xn =][X;, 


is the set of all n-tuples (x1,...,2,) with 2; € X;. 

To any set A C X we associate its characteristic function x 4 defined to take the value 1 in A 
and the value 0 in X\A. We also define its indicator function indf 4, which takes the value 0 in 
A and the value +00 in its complement. 


9.2.2.4 Graphs, epigraphs and hypographs 
Definition 9.2.2. To any mapping f : X — Y we define its graph: 


graph(f) = {(z,y); f(x) =y} CX x, 


a subset of the Cartesian product X x Y. 


Definition 9.2.3 To any function f : X — Ru, we associate its epigraph 


epi(f) = {(z,t) € X x Ry; t2f(z)}; 


(9.8) 
and its strict epigraph 
ePigtrict(f) = {(z,t) © X x Ry; t > f(x)}. 
(9.9) 
We define its finite epigraph as 
epifmite(f) = { (x,t) € X x R; t>f(x)}; 
(9.10) 


and its strict finite epigraph as 


CPi, ictinte (f) = {(z,t) © X x R; t > f(z)}. 
(9.11) 


The first two being subsets of the Cartesian product X x R, the following two of the product 
XxXR. 


It is easy to pass from the finite epigraph to the strict finite epigraph as well as in the other 
direction: 


CPi, scefinite (ff) = Uepin( fro); epi'mte( f) = 1 ePigericn( f—c). 
(9.12) 


Similarly for epi,,,;-; and epi. 
The finite epigraph of a function f : R” — R, is contained in the closure (taken for the usual 
topology in R” x R) of epi, ,finie (f), maybe strictly. 


Definition 9.2.4. Analogously we define the hypograph of a function: 
hypo(f) = {(z,t) € X x Ry; t<f(2)}, 
(9.13) 


as well as the strict hypograph, hy postrict(f), the finite hypograph, hypo!™*e( f), and the strict 
finite hypograph, hy po'™:**(f). 


It is often convenient to express properties of mappings in terms of their epigraphs or 
hypographs. 


9.2.2.5 Inverse and direct images 
To any mapping f : X — Y we define two mappings on a higher level, 
f : AY) — A(X) andf«: A(X) > AY). 
The first is defined by 
f° (B) = {x € X; f(z) € B}, Be AY). 
(9.14) 
Here f (B) is called the inverse image of B. The second is defined by 
fx(A) = {f(z); 2 € A}, Ace A(X). 
(9.15) 


The set f(A) is called the (direct) image of A. We write Im (f), the image of f for f+(X). 
The mappings f and f, are the simplest examples of the pullbacks and pushforwards used in 
differential geometry, and similarly in homology theory and distribution theory. 


Definition 9.2.5 To any function f : X — Ry, we associate its effective domain, denoted by 
dom(f) and defined as the set where the function takes values less than +00: 


dom(f) = {x € X; f(x) < +00} = f*({-00, +oo)), 


the inverse image of R U {—oo}. 


9.2.3 Defining convex sets 


It is most convenient to define convex functions with the help of convex sets. This also has the 
advantage that we can treat functions with infinite values without difficulty. That is why we start 
now with convex sets. 


Definition 9.2.6. A rectilinear segment in a vector space E is the set 
[a, b] = {(1 —t)a+ tb; t € R, 0<t<]}, 


where a and b are its endpoints. A subset A of a vector space is said to be convex if 
{a,b} c Aimplies [a,b] Cc A. 


Every segment, every straight line, and all affine subspaces are convex sets. 

It is worth noticing that to check convexity of a set, only its intersections with one- 
dimensional subspaces need to be considered. In other words, a set A C FE is convex if and only 
if the inverse image f *(A) is an interval (bounded or unbounded) in R for every mapping 
f:R- Eof the form f(t) = ta+b,te R,a,be EL. 

A convex set A in R” need not have interior points, but it does so if we consider it as a subset 
of the smallest affine space that contains it. We define the relative interior of A, denoted by 
relint(A), as the interior taken with respect to the topology in this affine subspace that is 
induced by the usual topology on R”. In this way every convex set, even if only a singleton set, 
has a nonempty relative interior. The set A \relint(A) is the boundary of A taken in the smallest 
affine space containing A and is more interesting than the boundary of A if the space mentioned 
is not all of R”. 


9,2.3.1 The convex hull 


Definition 9.2.7 Given a subset A of a vector space E, we define the convex hull of A as the 
smallest convex set containing A. It will be denoted by evxh( A). 


The convex hull is well defined since any intersection of convex sets is convex. 


9.2.4 The Hahn-Banach theorem 


Among the affine subspaces we will pay attention to the hyperplanes, those of codimension 1, 
which means that they are defined by a single equation (x) = c, where € is a nonzero linear 
form on the vector space. Other important sets are the half-spaces, which are defined by an 
inequality €(x)>c or (x) >. 

A topological vector space is a vector space E equipped with a topology such that both 
addition 


Ex Hx(tz,y)He+yEek 


and multiplication by scalars 


Cx Es(t,z)H tr EF 


are continuous. 

After having restricted some variables, we see that all translations x ++ x + a are continuous 
as well as all mappings t +> ta for a fixed a. The latter property implies that the inverse image of 
an open set in E under the mapping t +> ta is open for the usual topology in C. Similarly of the 
field of scalars is R. 


Theorem 9.2.8. (The Hahn—Banach theorem) Every open convex set in a topological vector 
space E is the intersection of a family of open half-spaces. 

Every closed convex set in a topological vector space is the intersection of a family of closed 
half-spaces—and also the intersection of a family of open half-spaces. 


There are topological vector spaces with dual equal to zero. The separation can then seem like a 
paradox. It is resolved by the fact that in these vector spaces, the only open convex sets are the 
empty set and the whole space. 

Here we shall accept this important theorem and refer to its proof in any of the many books 
on functional analysis. 

Werner Fenchel (1952) characterized the sets that are intersections of a family of open half- 
spaces and called them evenly convex . As the Hahn—Banach theorem states, all open convex 
sets, and all closed convex sets are evenly convex. So are all strictly convex sets and a set like the 
closed triangle in R? with one vertex removed defined by 0< 414+ %2<1, 4120, r2>0. An 
open triangle with one boundary point added is not evenly convex. See Section 9.7, page 326 for 
the definition of refined half-spaces, which serve to represent general convex sets. 


9.2.5 Supporting hyperplanes 


Definition 9.2.9. Given any set A in a topological vector space E, a supporting hyperplane is a 
hyperplane Y such that A is contained in one of the closed half-spaces defined by Y and such 
that the closure of A meets Y. 


Theorem 9.2.10 Let A be a convex subset of a topological vector space E with) # A # E and 
let a be any boundary point of A. Then there exists a supporting hyperplane of A passing through 
a. 


Proof In view of the Hahn—Banach theorem, there is a closed hyperplane Y that passes through a 
and such that A is in one of the closed half-spaces defined by Y. 


A set A in a topological vector space is said to be bounded if for any neighborhood U of the 
origin there exists a number Ay such that AU contains A for all A with |A|>Ao. 


Corollary 9.2.11 Let A be a bounded nonempty subset of a topological vector space and let a be 
any boundary point of cvxh(A). Then there exists a supporting hyperplane of A passing through 
a. 


Proof We apply the theorem to cvxh(A). A supporting hyperplane of cvxh(A) must also be a 
supporting hyperplane of A. 


9.2.6 Defining convex functions 


Definition 9.2.12 A function f : E — Ry, defined in a vector space E is said to be convex if its 
finite epigraph is convex as a subset of E x R. We shall write CV X(E, R1) for the set of these 
functions. 


The functions that take one of the values —oo, +00 identically are convex, since their finite 
epigraphs are, respectively, the whole space # x R and the empty set. An affine function is 
convex, since its finite epigraph is a closed half-space. 

It is easy to see that a function f : HE — Ry is convex if and only if it satisfies Jensen's 
inequality 


f(l—-t)e+ty)<(1—-4)f(z)+tf(y), 2 ye, O<t<l. 
(9.16) 


Here we define 0 - (+00) = 0-(—oo) = 0 (or else consider only 0 < t < 1). There are more 
general inequalites to be described now. 


9.2.7 Strict and strong convexity 


Definition 9.2.13 We shall say that a set A in a vector space is strictly convex if every supporting 
hyperplane cuts its closure in only one point. This means that the boundary of A does not contain 
any intervals of nonzero length. A function is said to be strictly convex if its finite epigraph is 
strictly convex. 


Definition 9.2.14 A function f :R" — Ry is said to be strongly convex if for every point 
a € E there is a positive number s such that x ++ f(x) — s||zx||3 is convex in a neighborhood of a 
(we use a Euclidean norm here). 

A subset A of finite-dimensional vector space E is said to be strongly convex if for every 
point a€ E there is strongly convex function f so that A agrees with the set 
{x © R"; f(@1,..-,&n—-1) < 2p} near a for some choice of coordinates in E. 


The function RSz +> V1+ 2? is strongly convex, but we see that the number s cannot be 
chosen independently of a. 
The function RSz +> 2* is strictly convex but not strongly convex. 


9.2.8 The convex envelope 


Definition 9.2.15 Given a function f : A— Ri, where A is a subset of a vector space E, the 
largest convex function F : FE — R, such that F'| ,4<f is called the convex envelope of f and will 
be denoted by evxe(f). 


Remark 9.2.16 In general we have 


(cvxe(f))(x) = inf [t; (a, t) € evxh(epi*™*¢(f))]. 


(9.17) 


The convex envelope of f : E —> Ry evaluated at a point x is equal to the infimum of all 
expressions 


N 
So Aj f(a + a), 


j=l 


where the j; and the a\) satisfy d;>0, S> A; = 1 and S> ,a\) = 0. We note that the points 
(c +a), f(a +.a")) belong to epi*™**(f), implying that the point 


(Sle+ 0), Do avsle+ a) 


belongs to cvxh(epi‘™'*e(f)) and that therefore (x,(evxe(f))(z) +t) belongs to 
cvxh(epi'™*(f)) for every positive number t. 


Remark 9.2.17 We have inclusions 


CPi, icine (cvxe(f)) = evxh(epif@te(f)) 
finite 


C evxh(epi(f)) c epif™*e(cvxe(f)). 


(9.18) 


The two inclusion relations here can be strict. 


Definition 9.2.18 Let A be any subset of a vector space E and let f : A — Ry be any function 
defined on A. We shall say that f is convex extensible if it is the restriction to A of a convex 
function defined on all of E, i.e., if there is a convex function F : E — R, such that F|, = f. 


There may exist more than one convex extension of a given function f For example 
F(a) = |x| — land F’* (x) = max(|z| — 1,0), x € R, have the same restriction to A = Z\{0} 


. If there exists a convex extension, then cvxe(f) is the largest one. 


9.2.9 Normed spaces 


To any vector space E over the field of real numbers, we associate its algebraic dual E*, the 
vector space of all linear forms on £&, ie., the functions €:&H—R_ satisfying 
&(a + ty) = €(x) + t&(y) for allz,y € Eandallt ¢ R. 

We say that a function BDz +> ||2|| ¢ R is a nommif ||x||>0 with equality if and only if 
xz =0; |jz+y||<||z|| + |ly|| for all x and y; and finally ||tx|| = ¢||z|| for all 2 € E and all 
positive numbers ¢. This means that the function E?3(z, y) > ||x — y|| is a metric with the extra 
property of being positively homogeneous. The subadditivity and the homogeneity together 
imply that x ++ ||z]| is convex. 

The space R” of all n-tuples can be normed by the /? norm || - ||p, 1<p< + 00, which is 
defined for l<p < +00 by 


i 
lele=(SckesP) = (ey. an) ER”. 


When p = +00 this has to be interpreted as a limit. More explicitly one defines 


|Z loo = max,|ax,|, z Ee R”. 


In addition to the algebraic dual E*, which is defined for any vector space, we consider for 


any normed vector space E also its dual space, denoted by E and consisting of all continuous 
linear forms on E. These are the linear mappings € : H — R such that |€(x)|<C||z|| for some 
constant C. On the dual we define the norm dual to || - || by 


Il = sup €(z)}|, €€ E’. 


I|z||<1 


(9.19) 


It follows that |€(x)|<||€||'- ||z|| for all 2 € Hand all € € E’. 


When E = R”, we may identify both E* and E with R”, and the evaluation of € at the point 
x, i.e., the number €(zx), is then the inner product, defined by €- x = 121 +--+ + &n2%n. The 


Euclidean norm || - ||2, defined by ||2||? = a - a, is dual to itself: 
Nels = sup €(z) = lélle = / 08. 
|e \|2<1 
It is not difficult to prove that the norm dual to || - ||1 is || - ||. and vice versa. More generally, 


one can prove that the norm dual to || - ||p is || - ||~, where g = p/(p — 1), 1 < p < +00, witha 
natural interpretation also when p = 1, +00. This statement follows from Hdlder's inequality and 
its converse. 

A vector space provided with a Euclidean norm is called a Euclidean space. 


9.2.10 Duality in convex analysis 


By the term duality we aim at properties and results that involve a vector space and its dual. The 
most important examples are the support function and the Fenchel transformation, to be defined 
now. 


9.2.10.1 The support function 


Definition 9.2.19 Given any subset A of a vector space E we define its support function H 4 by 


H4(£) = supé(z), Cee”, 


rEA 


Here E™* is the algebraic dual of E. 


Example 9.2.20 If A is a ball, A= B<(c,r) with r>0, or Be(c,r) with r> 0, then 
H4(€) = €(c) + r||€||', € € E*, where || - || is an arbitrary norm in E and || - \|' its dual norm, 
defined on E* in (9.19) above. 


9.2.10.2 The Fenchel transformation 
Definition 9.2.21 To any function py : E — R, we define its Fenchel transform ¢ by 


O(f) = sup(é(x) — v(z)), 


cee 


defined for  € E™, the algebraic dual of E. 


It follows that 


E(x)<p(z)+ Gl), eEeH, €¢E", 


(9.20) 
called Fenchel's inequality. 
It is evident that G is the smallest function g such that €(x)<y(a) + g(€) holds. 
For any family of functions (pj) 7, ; € F(E, R1), we clearly have 
sup@ = (infy)-. 
uD? (infy) 
(9.21) 


We see that the support function of a set is the Fenchel transform of its indicator function: 


AH, = indf,. 


Example 9.2.22 If the graph of a function 9 is a paraboloid, p(z) =a+ B-x + +ellx||}, 
z € R”, wherea € R, 8 € R” andc > 0, then the same is true of the graph of its transform: 


G(E) = —-a+t Fe *||E — All}. 


We now ask what happens if we apply the transformation again. We let = be any nonempty 


subset of E*, and define the Fenchel transform of any function fdefined on = by 


f(z) = sup(&(z)-— f(), eC E. 


6e8 


Here we can take 5 = {0} as well as 5 = E*. Ina normed space it is customary to take 5 = E’ 
, the dual of E. 
The equivalence 


@<fif and only iff<y, y € F(E,R)), f ¢ A7(Z,R)), 
(9.22) 


follows easily from the definition. 


We may form the second transform G of a function defined on FE. The main result in the 
theory of the Fenchel transformation is the following. 


Theorem 9.2.23. (Fenchel's theorem) Let y be a function defined on a vector space E and let £ 


be any nonempty subset of its algebraic dual E*. Then we always have @<.p. Equality holds if 
and only if 


(A) 9 is convex; 


(B) @ is lower semicontinuous for the weakest topology for which all linear forms in = 
are continuous; and 


(C) does not take the value —oo unless it is identically equal to —oo. 


Property (B) here means that if y(a) > s, then there are linear forms €1,...,€m in = and a 
number 8 > 0 such that v(x) > s when |€;(% — a)|<0, j = 1,...,m. We shall denote this 
topology by o(£, =). (We get the chaotic topology when © = {0}, implying that the only lower 
semicontinuous functions are the constants.) 

In R” we usually choose = = R”; the semicontinuity is then semicontinuity with respect to 
the usual topology of R”. 

Before we go on, let us consider other expressions of this semicontinuity. The topology 
o(E, =) on E gives rise to a topology in E x R, viz. the product topology, for which a basis for 
the neighborhoods of a point (a, s) € E x R are given by the sets 


1(e0)s (a= a) <0.9 = Ty Ns |= 89, €&,€5, @>0. 


Let us call this topology 7(E x R, =). 
So a function @ that satisfies the three conditions can be represented as the supremum of a 
family of affine functions y(x) = sup(€(x) — @(€)). This can be most helpful in proving that 
g 


certain functions are convex. 
We accept Fenchel's Theorem without proof here. 
It is now clear that Fenchel's inequality can be improved to 


&(x)<G(x) + G(E). 


For Werner Fenchel's pioneering work on duality in convexity theory, see (1949, 1952, 1953, 
1983). 


9.2.11 Introduction to complex convexity 


Every vector space over the field of complex numbers is at the same time a real vector space, 
obtained by simply restricting the multiplication by complex scalars to real scalars. So everything 
what we have said here about convexity applies also to complex vector spaces. But the presence 
of complex numbers gives birth to new phenomena. Let us list here several variants of complex 
convexity, in increasing order of strength. Most of them will be considered in detail in later 
sections of this chapter, some of them in other chapters. 


1. An open set 0 in C” is pseudoconvex if there is a continuous plurisubharmonic function 
defined in © that tends to +00 at the boundary of 1. See, e.g., (Hérmander 1990: Theorem 
2.6.7 and Definition 2.6.8.) 


2. An open set 0 in C” is a domain of holomorphy if there exists a holomorphic function 
defined in Q that cannot be continued, in a precise sense, over the boundary of 0. See, e.g., 
(H6rmander 1990: Definition 2.5.1). 

3. A connected open set in C” is called hyperconvex if there is a continuos negative 
plurisubharmonic function u defined in Q such that the sublevel set {z € Q; u(z)<c} is 
compact for every negative number c. See, e.g., (Kerzman & Rosay 1981). 


There are several different notions of convexity related to lineal convexity. In increasing order of 
strength we have: 


1. Local weak lineal convexity in the sense of Yuzakov & Krivokolesko, see Definition 9.5.8 
on page 303; 
2. Local weak lineal convexity; see Definition 9.5.7 on page 302; 


3. Weak lineal convexity, originally introduced as Planarkonvexitat by Behnke & Peschl 
(1935:158,162); see Definition 9.5.3 on page 301; 


4, Lineal convexity, introduced as convexité linéelle by André Martineau (1966: 73; 
1977:228); see Definition 9.5.1 on page 300; 

5. C-convexity, originally introduced as convexité linéelle forte by Martineau (1967:400; 
1968; 1977:265,325). 
H6érmander (1994: Definition 4.6.6) defines an open susbset of C” to be C convex if 20 L 
is a connected and simply connected subset of L for every affine complex line L. 
Andersson, Passare & Sigurdsson (2004: Definition 2.2.1) first defines a subset E of P to be 


C -convex if EF # P and both E and its complement P\F are connected. A subset E of P” 
is called C -convex if all its intersections with complex lines are C-convex. 


6. Then we have the usual convex sets already discussed; see Definition 9.2.6) on page 256. 
7. The strict convex sets; see Definition 9.2.13 on page 258. 


8. The strong convex sets; see Definition 9.2.14 on page 259. 


9.2.12 Notes on the history of the concepts discussed in this chapter 


I learned about lineal convexity from André Martineau during the academic year 1967-1968 
when I was in Nice with him. His premature death on 1972 May 04 was a great loss to world 
mathematics. He introduced also the notion of strong lineal convexity (1968), which, however, 
was not geometrically defined. Later Znamenskij (1979) found a geometric characterization; the 
property is now called C-convexity. Nowadays the most important sources for C-convexity are 
the book by Hérmander (1994) and the survey by Andersson, Passare & and Sigurdsson (2004). 
My earlier contributions to the field are to be found in (1978, 1996, 1997, 2016, 2019). 


9.2.13 A note on terminology 


Heinrich Behnke and Ernst Peschl (1935) introduced the notion which is now known as weak 
lineal convexity. They called it Planarkonvexitat. 

André Martineau used the terms convexité linéelle and linéellement convexe—see Martineau 
(1966:73) and (1968:427), reprinted in (Euvres de André Martineau 1977:228) and (1977:323), 
respectively. In French there are two adjectives, linéaire, corresponding to the English Jinear, and 
linéel, which I rendered as lineal. (There is also an adjective Jinéal.) Martineau obviously wanted 
a distinctive term in order to signal the special meaning of his convexity, not to be misunderstood 
as ordinary convexity. Diederich & Forness (2003) and Diederich & Fischer (2006) write 
“lineally convex.” 

In Russian, the adjective “Ue js most often used for both French terms Jinéel and 
linéaire, and this is the term used by Aizenberg, Krivokolesko, Yuzakov, and others who write in 
Russian. In the translations into English of these Russian texts, there appears most often linear 
convexity and linearly convex. 


Also Znamenskii (1979:83; 1990:1037) used aunetinwi, as did Znamenskii & Znamenskaya 
(1996:359). 
Later Znamenskii (2001) used aunetivammi (usually translated as ‘ruled’; a common term is 


aunetiuamoii ‘ruled surface’). He thus established the distinction between Jineal, linéel and Iinear, 
linéaire in Russian (Yurii Zelinskij, personal communication 2013 March 26). 

Hormander (1994:290, Definition 4.6.1), Andersson, Passare & Sigurdsson (2004:16, 
Definition 2.1.2), and Jacquet (2008:8, Definition 2.1.2) used Jinear and linearly and thus did not 
keep the distinction introduced by Martineau. In my opinion, these authors unnecessarily copied 
the usage in the translations from Russian and did not pay attention to the pioneering work of 
Martineau. It should also be noted that the English Jineal is actually the older of the two words, 
being attested since the fourteenth century, while Jinearis attested from 1706 (Webster 1983). 

Another term is hypoconvex. The first appearance in this context? that I have found is in 
Helton & Marshall (1990:182), where it is used for sets with a boundary of class C and has the 
meaning of ‘strongly lineally convex’ (satisfying the strict Behnke—Peschl differential condition 
as I called it in my paper (1998:3); later it was weakened to a synonym of lineally convex by 
Whittelsey (2000:678), and used in this sense by Agler & Young (2004:379). The term helpfully 
reminds us that it signifies a property weaker than convexity. 


2Norberto Salinas (1976:144, 1979:327) used the term hypoconvex in a different sense. 


Dr 


9.3 Introduction to Mathematical Morphology 


9.3.1 Introduction to this section 
Lattice theory is a mature mathematical theory thanks to the pioneering work by Garrett Birkhoff 


(1911-1996), @ystein Ore (1899-1968), and others in the first half of the twentieth century. A 
standard reference is still Birkhoff's book (1995), first published in 1940. 

Mathematical morphology is a branch of science that was created in the 1960s by Georges 
Matheron (1930-2000) and Jean Serra. It thrives in complete lattices. 

Mathematical morphology can be described as lattice theory applied to several branches of 
science, in particular, to image analysis and image processing, where many concepts and 
procedures can be successfully described with concepts from mathematical morphology. In this 
section we give the most basic definitions and the simplest properties—those that will be useful 
in the coming parts of the chapter. For more complete treatments, see the books by Matheron and 
Serra mentioned in the list of references as well as my papers (2007, 2010) and my book 
manuscript (ms 2021). 

With the arrival of tropical geometry, lattice theory can (now) be viewed as a tropicalization 
of other mathematical theories. Developments originate in several branches of mathematics, for 
instance algebra (Blyth & Janowitz 1972, Blyth 2005), logic (Stoltenberg-Hansen et al. 1994, 
Gierz et al. 2003), general topology and functional analysis (Gierz et al. 2003:xxx—xxxil), 
convexity theory (Singer 1997), and, for mathematical morphology with applications in image 
processing, books by Matheron (1975), Serra (1982), Serra, Ed. (1988), and Heijmans (1994); 
articles by Heijmans & Ronse (1990), Ronse (1990), Ronse & Heijmans (1991, 1998), Heijmans 
(1995), Serra (2006), and Ronse & Serra (2008, 2010). Other areas where concepts from lattice 
theory are used include semantics (abstract interpretation) of programming, the theory of fuzzy 
sets, fuzzy logic, and formal concept analysis (Ganter & Wille 1999). For general lattice theory a 
standard reference is Gratzer (1998). 

This variety of sources for fundamental concepts has led to varying terminology and hence to 
difficulties in tracing history. 

The two concepts of lattice and complete lattice must be carefully distinguished. This 
becomes obvious when we see that a complete lattice L can contain another complete lattice M 
with M as a sublattice of L ... but Mis not a sub-complete-lattice of L (see Example 9.3.15). 

A useful tool in the sequel will be generalized inverses and generalized quotients. They come 
in two versions, lower and upper. 

To define an inverse of a general mapping seems to be a hopeless task. However, if the 
mapping is between preordered sets, there is some hope of constructing mappings that can serve 
in certain contexts just like inverses do. 

There is an analogy between lattice theory and the theory of vector spaces. The theory of 
topological vector spaces was developed to a large extent because of the theory of distributions, 
which in turn was motivated by applications in partial differential equations. Developments in 
image processing motivated a renewed interest in lattice theory, in particular in complete lattices. 


Lattice theory was applied to switching circuits, and it was then enough, because of general 
finiteness conditions, to form models using lattices, but in image processing it is more convenient 
to assume completeness; for a motivation, see (Ronse 1990). 

While vector spaces are useful in modelling linear problems, lattices seem to be more 
adapted to nonlinear problems. Auditory phenomena are often additive: all the instruments of an 
orchestra can be heard, while with visual phenomena this is not so: one object can block another 
from our view. This indicates that linear models may suffice for the first kind of phenomena 
(Fourier analysis and synthesis are successful for sound waves), while the visual ones are more in 
agreement with nonlinear operators, like maximum and minimum. 

There are also analogies between topological spaces and preordered sets, in particular 
lattices. The continuous linear mappings in the first case correspond to increasing mappings in 
the second. 

A comparison of the equations a+ 2 = 6b and aV x =b shows that the second is more 
complicated than the first: The first has the unique solution z = b— a for a,b € R, while the 
second has no solution if a,b € R with a > b; a unique solution x = 6b if a < 6 and infinitely 
many solutions x<a if a = b. In our schools we tend to prefer problems with a unique solution 
but in real life problems are more like a V x = b. 


9.3.2 Preorders and orders 


For the morphological operations on the computer screen, we need to consider families of sets. 
The family of all subsets of a given set is ordered by the inclusion relation, which is an example 
of an order relation. It is therefore convenient to introduce concepts that will be useful in the 
general theory of order relations. In this subsection we shall do so. 


Definition 9.3.1 A preorder in a given set X is a relation (a subset of X x X = X?) which is 
reflexive and transitive. A preordered set is a set together with a preorder. 
An order is a preorder which is antisymmetric. An ordered set is a set together with an order. 
Two elements x and y are said to be comparable if either x<y or y<x. 
An order is said to be total if any two elements are comparable. 


The definition of preorder means, if we denote the relation by <, that 
La, x € X; and that 
(9.23) 
x<yand y<zimplies r<z, L,y,z2 Ee X. 
(9.24) 
The definition of order means that the relation shall in addition satisfy 
roy yor> r=y, x,ye X. 


(9.25) 


We shall write x < y if x<y and x # y. We shall also write x>y and x > y for y<a and 
y < Z, respectively. 

As already mentioned, a basic example of an ordered set is the power set of a set W with the 
order relation given by inclusion, thus A<B being defined as A C Bfor A, BE A(W). 

Suppose that we have two preorders defined in a set X; denote them by < and x. The 
preorder < is said to be finer than the preorder <, and < is said to be coarser than <, if x<y 
implies x~y for all x,y. 

There is a finest preorder in a set, viz. when we define x<y to mean that x = y. This preorder 
is an order; let us call it the discrete order . There is also a coursest preorder in any set X, when 
we declare that x<y for all x,y € X. Let us call this the chaotic preorder . The set of all 
preorders on any set is thus an ordered set with a largest and a smallest element. 

In a preordered set L, given a, b € L, the interval[a,b] is the set 


{x € L; a<x<b}, 
in particular to be used when L = Ry. We shall write [a, b], for an interval of integers, thus 
[a,b], = [a, |] NZ = {a € Z; axax<bd}, a,b € Z. 
(9.26) 


Definition 9.3.2 An equivalence relation is a preorder which is symmetric, 1.e., such that x=<y if 
and only if yX2. 


This means that x<y implies y<z for all z,y € X. 


9.3.3, Mappings between preordered sets 


In preordered spaces the increasing mappings are of importance: 


Definition 9.3.3 If f : X — Y is a mapping from a preordered set X to a preordered set Y, then 
we say that f is increasing if 


forall z,z' € X, therelation x< xz’ implies f(x)<yf(z’'). 


We shall write incr(X, Y) for the set of all increasing mappings X > Y. 
We shall say that f is decreasing if 


therelation x< xx’ implies f(x)>y f(z’) 


for all elements x, x' € X. 


The increasing mappings play the same role in the context of ordered sets as the linear 
mappings in the theory of vector spaces and as the continuous mappings in the theory of 
topological spaces. 

A preorder < is finer than another preorder < if and only if the identity mapping 
(X,<) > (X, =) is increasing. 


If f,g: X — Y are increasing, then so are f \ g and f V g. If alsoh: Y — Z is increasing, 
then so is h°f: X — Z. In particular, when X =Y, we have the three operations 
(f,9)0 fAg,f\ 9,9 f, which all preserve the property of being increasing. 

A comparison with topology is in order here. If f : X — Y is a mapping of a topological 
space X into a topological space Y with topologies (families of open sets) ty and ty, we can 


define a new topology t-in X as the family of all sets 


{xz € X; f(x) € V}, V aay: 


Then fis continuous if and only if ry is finer than Tt» 


For mappings f : X — X with target set equal to the domain we can form the iterations f° f, 
f° f° f and so on, and among these mappings those that satisfy f° f = f are of interest: 


Definition 9.3.4 We shall say that a mapping f : X — X is idempotent if f° f = f, ie., if 
f(f(x)) = f(x) for all x © X. A mapping which is both increasing and idempotent will be 
called an ethmomorphism. 


Definition 9.3.5 We shall say that a mapping f : X — X is extensive if it is larger than the 
identity, i.e., f(x)>a for all elements x € X. We shall say that it is antiextensive if it is smaller 
than the identity, i.e., f(x)<a forallxz € X. 


We define the invariance set of a function f : X — X as the set of all x € X such that 
f(x) = ax. We denote it by invar(f). For extensive mappings f the invariance set is decreasing 
in f while it is increasing in ffor antiextensive mappings. 


9.3.4 Cleistomorphisms and anoiktomorphisms 


Definition 9.3.6. A cleistomorphism in an ordered set X is an ethmomorphism (see Definition 
9.3.4): X — X which is extensive (see Definition 9.3.5); in other words, which satisfies the 
following three conditions. 


z<a’' implies K(x)<K(z’); 


(9.27) 
K(K(x)) = K(2); 
(9.28) 
CR A(@); ee xX 
(9.29) 


for all elements x, x' € X. 


The element K(x) is said to be the closure of x. Elements x such that K(x) = a are called 
invariant or closed (for this operator). An element is closed if and only if it is the closure of some 


element (and then it is the closure of itself). 
In many applications the set X is the power set A(W) of some set W. Then the 
cleistomorphism is given as an intersection: 


A= OY Y isclosedand Y > A). 


When W is a topological space, a basic example is the topological closure operator which 
associates to a set in a topological space its topological closure, i.e., the smallest closed set 


containing the given set, denoted by A+> A. In fact a cleistomorphism in A(W) defines a 
topology in W if and only if it satisfies, in addition to (9.27), (9.28), (9.29) above, two extra 
conditions, viz. 


0=OandAUB=A UBforallA, BC W, 


(9.30) 


where © denotes the empty set. 
Another cleistomorphism of great importance is the operator which associates to a set A in 
R” its convex hull, the smallest convex set containing the given set, denoted by cvxh(A). The 


composition A ++ evxh(A) is a cleistomorphism, whereas the composition in the other order, 
At cvxh (4 ) is not idempotent if 22. We see that the composition of two cleistomorphisms 


is sometimes, but not always, a cleistomorphism. 
Dual to the concept of cleistomorphism is the concept of anoiktomorphism. 


Definition 9.3.7 An ethmomorphism a: X — X is said to be an anoiktomorphism if it is 
antiextensive; in other words, if it satisfies the following three conditions. 


xz<za' implies a(x)<a(z'); 
aa=a; 
a(xz)<a, 


for all elements x,x' € X. 


The composition of a cleistomorphism and an anoiktomorphism is always idempotent: 


Proposition 9.3.8 Let a,«:L-— L be an anoiktomorphism and a cleistomorphism. Then 
7 = a°k and@= k a are ethmomorphisms. 


Proof That 7 and @ are increasing is obvious. Since x is extensive, we get 
WN=AKAKFAAK=AK=7. 
Since a is antiextensive, we get 


WN=AK AKA K K=Aa°K= YN, 


so 7 is idempotent. The proof for @ is similar. 


Example 9.3.9 A typical example is when we take a and x as the operations of taking the interior 
and the topological closure in a topological space, respectively; a(A) = A , «(A) = A. Thena 
fixed point of the composition a’ « is called a regular open set and a fixed point of ka is called 
a regular closed set. These operations are neither extensive nor antiextensive in general. 


Proposition 9.3.10 The infimum of a family of cleistomorphisms in a complete lattice L is a 
cleistomorphism. The supremum of any family of anoiktomorphisms in L is an 
anoiktomorphism. 


Proof Let Kj j € J, be cleistomorphisms and define K = /j<-7«;, meaning that the value of x at 


z € L equals \j<yK;(x). Clearly « is increasing and larger than the identity. It follows that 
kK K>k. To prove the opposite inequality, we note that K°K<K;°K; = K;. Taking the infimum 
over all j we get what we want. 

The result for anoiktomorphisms follows by duality. 


9.3.5 Lattices and complete lattices 


Let L be an ordered set and A a subset of L. An element b € L is said to be the infimum of all 
elements a € A if bis the largest minorant of all a € A. This means that b<a for all elements 
a € A, and that if b’<a for all a € A, then b’<b. The infimum, if it exists, is necessarily unique. 
The infimum of the empty set exists if and only if L possesses a largest element, and if so, the 
infimum is this largest element. 

We shall write 


b= infa = inf(a; a € A) = Aaeaa 
acA 
for the infimum of all elements in A; if A has only n elements we write b = a; A --- A Gy. If the 
infimum belongs to A, we call it a minimum. As an example, the set of all positive real numbers 
has 0 as its infimum, but 0 is not a positive number and therefore not a minimum. 
Similarly we define the supremum 


c = supa = sup(a; a € A) = Vacaa 
acA 


as the smallest majorant of all elements in A. If the supremum belongs to A, we call it a 


maximum . The supremum of all elements in the empty set, supa, exists if and only if L has a 
red 
smallest element. 


Definition 9.3.11 Let L be a nonempty set. If any subset consisting of two elements in L has an 
infimum, we shall call L an inf-semilattice; similarly, if any two-set of L has a supremum, we 
shall call L a sup-semilattice. If L is both an inf-semilattice and a sup-semilattice we shall call L 
a Jattice. 


Definition 9.3.12 If any nonempty subset (finite or infinite) of a nonempty set L has an infimum, 
L will be said to be a complete inf-semilattice; analogously we define complete sup-semilattice 
and complete Iattice. 


We may denote the smallest element in a complete lattice by 0 and the largest by 1. We have 


supz = infe=1; infx = supz = 0; 
rel reo re re 


the infimum of the empty set exists and is 1, and the supremum of the empty set is 0. 


A sublattice is defined just like a subgroup with respect to the operations /\ and V: that Mis 
a sublattice of L means that for all z,ye M, xAy and x Vy, when calculated in L, are 
elements of M. A sublattice is therefore something more than a subset with the induced order; 
see the following examples. 


Example 9.3.13 The space of real-valued continuous functions on a topological space is a lattice 
with the usual order: f<g if and only if f(x)<g(x) for all x. The space C'(R",R) of real- 
valued continuously differentiable functions on R” is not a sublattice of C(R", R) ifn=1. It is 
not even a lattice on its own. (The functions R>x +> Vt? + 2, t > 0, converge to x +> || as 
t — 0, but there is no infimum in C'(R, R).) 


Example 9.3.14 The family A(W) of all subsets of a set W is a complete lattice, with 
AA; =NA,; and VA; =UA;. The compact sets in R” form a sublattice 4(R") of A(R"). 
This lattice is a complete inf-semilattice but not a complete sup-semilattice. The family 
Move Ee )s n>1, of all convex compact sets is a lattice but not a sublattice of KH (R"): the 
supremum of two convex compact sets is not always the same in the two lattices. 


Example 9.3.15 The family of all closed sets in R", denoted by @(R"), is a sublattice of 

£(R"): the union and intersection of two closed sets are closed. But, although @(R") is a 
complete lattice, it is not a sub-complete-lattice of the complete lattice A(R”) when n>1. The 
union of a family of closed sets is not always closed, but there is a supremum, viz. the closure of 
the union. Thus, finite suprema agree with those in A(R") while infinite suprema do not. 


Example 9.3.16 The set F(R", R,) of all functions defined on R" and with values in the 
extended real line R, is a lattice under the usual order for real numbers, extended in an obvious 
way to the two infinities. The subset of all convex functions is ordered in the same way and is 
also a lattice under this order. However, the convex functions CVX(R",R,) do not form a 
sublattice of F(R”, R:) if n>1. The supremum of two convex functions is equal to the 
pointwise supremum of them: 


(f V g)(@) = f(z) V g(x) = max(f(z), 9(2)), 


but the infima are different in the two lattices: the infimum in the lattice of convex functions is 


f Acvr 9 = sup(h € CVX(R”, R1); h<f,9)<f A g = min(f, 9), 


where the supremum of all h<f,g is calculated in F(R", R1) and has a sense because that 
lattice is complete. That the two infima may be different is shown by easy examples like 


f(x) =e", g(x) =e-*, x CR. Here min(f, 9)(x) =e |*|, while f Acux g = 0. 


9.3.5.1 Dilations and erosions in complete lattices 


Mappings of the form Y(R”)5A+> A+ Be A(R") with a fixed set B are called dilations. It 
can be proved that a mapping A(R”) > A(R”) which commutes with translations and the 
formation of infinite unions is necessarily of this form. In lattice theory it is therefore natural to 
take the latter property as a definition: 


Definition 9.3.17 We say that a mapping 6: L — M, where L and M are complete lattices, is a 
dilation if it commutes with the formation of suprema, 1.e., 


O(VreAr) => VreAO(2) 


for all subsets A of L. 


In particular we get 6(0;,) = Oy, (take A empty), while 
6(1z) = Veet O(4)<1y. 
(9.31) 


Definition 9.3.18 Similarly we shall say that ¢ is an erosion if it commutes with the formation of 
infinite infima, 


for all subsets A of L. 


We note that ¢(1,) = 1y (take A empty), while (07) = Azere(x) 20m. 

Dilations and erosions are always increasing. Indeed, we have 6(x V y) = 6(a) V d(y). If 
xr< zy, this equation simplifies to 6(y) = 6(x) V 6(y)>m6(ax), which shows that 6 is increasing, 
6 € iner(L, M). A similar argument shows that erosions are increasing. 

In R” or C” we define 43,¢€p: A(G)> A(G) by d6p(A)=A+B and 
€p(A) = {x € G; z+ BC A}. It is easily seen that 63(A) C Cif and only if A C ep(C). Ina 
lattice this may be written as 6(x)<y iff x<e(y), equivalently as epid = (hypoe), where the 
symbol ~ means that we swap the components: for a subset A of a Cartesian product X x Y we 
define 


A= {(y,x); (x,y) © A}CY xX. 
(9.32) 


May we use this as a model to define erosions from dilations and conversely in the more general 
lattice situation? Indeed this is the case, and we shall do so in the next subsection. 


Example 9.3.19 The mapping f° : A(Y) + A(X), defined by (9.14), is both a dilation and an 
erosion, while fx : P(X) — AY), defined in (9.15), is a dilation but in general not an erosion. 


9.3.6 Inverses and quotients of mappings 


9.3.6.1 Introduction 


In this subsection we shall study inverses and quotients of mappings between ordered sets which 
are analogous to inverses 1/y and quotients x/y of positive numbers. The theory of lower and 
upper inverses defined in Subsubsection 9.3.6.2 generalizes the theory of Galois connections as 
well as residuation theory and the theory of adjunctions. An interesting question is to what extent 
a generalized inverse can serve as a left inverse, as a right inverse, and how an inverse of an 
inverse relates to the identity mapping. These inverses and quotients can be used to create a 
convenient formalism for a unified treatment of dilations 6 : Z — M and erosions e: M — L as 
well as of cleistomorphisms & = ¢€°6 : ZL — L and anoiktomorphisms a = 6°¢: M > M. 

In general a mapping g : X — Y between sets does not have an inverse. If g is injective, we 
may define a left inverse wu: Y + X, thus with u°g = idx, where idy denotes the identity 
mapping in X, defining u(y) in an arbitrary way when y is not in the image of g. If g is 
surjective, we may define a right inverse v: Y — X, thus with g°v = idy. We then need to 
define u(y) as an element of the preimage {z; g(x) = y}. In the general situation this has to be 
done using the axiom of choice. In a complete lattice, however, it could be interesting to define 
u(y) as the supremum or infimum of all points x such that g(z) = y, even though this supremum 
or infimum need not belong to the set of points that are mapped to y. At any rate, the preimage of 
y is contained in the interval defined by the infimum and the supremum. However, for various 
reasons it is convenient to take instead the infimum of all x such that g(x) >y or the supremum of 
all x such that g(x)<y. This yields better monotonicity properties. (The case g(x) = y is 
allowed, since we can take the preorder in Y be the discrete order.) 

If, given a mapping g: L — M from an ordered set L into an ordered set M, we can find 
mappings u,v: M — L such that hypou = (epig) or epiv = (hypog), we would be content 
to have some kinds of inverses to g. However, usually the best we can do is to study mappings 
satisfying either hypou > (epig) or epiv > (hypog). This will be the approach in what 
follows, where we shall define not one but two inverses, viz. the lower (to be denoted by gj-1)) 
and the upper (written as g!-1]), 


9.3.6.2 Defining inverses of mappings 


Definition 9.3.20. Let L be a complete lattice, M a preordered set, and g : L —> M any mapping. 
We then define the lower inverse g;_,,: M — L and the upper inverse gl: M — L as the 
mappings 


gy (y) = Veex(2; 9(@)<uy)) = Veex(2; (x,y) € epig); 


(9.33) 


g N(y) = Aver (2; g(a)>my)) = Aver (2; (2, y) € hypog), 


(9.34) 


where y € M. 


As a first observation, let us note that these inverses are always increasing. If there exists a 
largest element 1), then 9{-1] (1a) = 11. Similarly, if M possesses a smallest element 0,4 then 


gl (Oyr) = Or. If M has the chaotic preorder, then both inverses are constant, 9-1) = 1, and 
gH = 0, identically. Here the lower inverse is larger than the upper inverse. 


Example 9.3.21 For any mapping f : X — Y we defined in (9.14) and (9.15) the mappings 
f°: AY) > A(X) and fe: A(X) > AY). We note that, for all AcE A(X) and all 
Bc AY) we have f(A) C B if and only if A C f*(B). It follows that fy = fx and 
(f+) a = f’. From this we see that (f°) ")_y —f and that ((f*)_))'" ee 


9.3.6.3 Properties of inverses 
We note that we always have 
epig C (hypogi-1)) 
(9.35) 


in other words, if y>g(a), then x<gj_1j(y); and 


hypog C (epig'")), 
(9.36) 


in other words, if y<g(x), then x>gl-(y). Here R for a subset R of X x Y is defined by 
(9.32). In general, these inclusions are strict. 


Example 9.3.22 If f: A(G) > A(G) is the dilation f(A) = 6y(A) =A+U, AE AG), 
where G is an abelian group and U a fixed subset of G, called the structuring element, then 


(60);-4(©) =U (4: &u(A) CC) =e0(0), Ce FO), 


the erosion associated to 6 yy. A most fundamental example. 
The compositions ay = dy°éy and ky =éy dy are the anoiktomorphism and _ the 
cleistomorphism associated to 6 y, respectively. 


When X = Y = R”, the upper inverse of 6 y is not interesting, since 


(6u) “(C) =0 


for all C if U has interior points, and equal to C ifU = {0}. 


An ideal inverse u would satisfy u°g = idz, g°’u = idy, and the inverse of u would be g. It is 
therefore natural to compare g)_4)°g and gi I°g with id;; g° gia) and g° gl with id, and 
inverses of inverses of g with g. We shall not do so here but refer the reader to the book 
manuscript (ms 2020). 


9.3.6.4 Quotients of mappings 


We shall now generalize the definitions of upper and lower inverses. 


Definition 9.3.23 Let a set X, a complete lattice M, and a preordered set Y, as well as two 
mappings f : X — M andg: X — Y be given. We define two mappings f/, 9, f/*g: Y + M 
by 


(f/+ g)(y) 
(f/*9)(y) (f(z); g(z)>yy), yeY. 


We shall call them the lower quotient and the upper quotient of f and g. 


= Vaex 
= Nex 


xX 

aoe. \/ 
Y ~~ M 
f/*9f/°9 


We shall often assume that X, Mand Y are all complete lattices, but this is not necessary for the 
definitions to make sense. 

The mappings f/, 9, f/*g € F(Y, M) are always increasing. 

The quotients f/, g and f/*g increase when f increases, and they decrease when g increases 
—Just as with the division of positive numbers: 


If f)<fo and g,;>ygo, then f,/, g;<fo/, g.andf)/*g,;<wfo/*go. 
If g(x)<yy, then f(2)<m(f/+9)(y); if 9(@)>yvy, then f(x) > u(F/"9) (y). In particular, 


if g(x)=y, then (f/"g)(y)<mf(x)<m(E/, g)(y)- 


We note some special cases. 

(1). If we specialize the definitions to the situation when X = M and f =idy, then 
f/«g =idx/.9 =g[-1) and f/*g =idx/*g = g\l; cf. Definition 9.3.20. So inverses are 
quotients. 

(2). A second special case is this: Taking Y = M and g = f in the definition we see that, for all 
mappings f : X — M we have 


f/, f<idur<f/*f; 
(9.37) 


Gtk Fafa UT) SF 
(9.38) 


9.3.7 Set-theoretical representation of dilations, erosions, cleistomorphisms, and 
anoiktomorphisms 


We present here an easy result for translation-invariant operators on the family of subsets of an 
abelian group, putting several operations under a common roof. 


Proposition 9.3.24. Let S be a subset of an abelian group G. Then the dilation, erosion, 
cleistomorphism and anoiktomorphism with structuring element S can all be written in the form 


ys,r.u(A) 1 U(z +S;24+TC A+ U), Ae P(G), 
rE 


(9.39) 

for special choices of the structuring elements S, T, U, viz. 

dg = Y5S,{0},{0}, ES = Y{0},S,{0}; AS = YS,5,{0}, KS = Yio},5,5- 
(9.40) 
We can also write the mappings as 
¢ = (f/+9)°h, 
where 
f(B) = BorB+S, g(B)=B+S, h(A)=A or A+S, 
for A, Be AG). 


Proof The dilation 6= ds, the erosion € = 6)_1j, the cleistomorphism « =e°d, and the 
anoiktomorphism @ = 6° can be written 


6(A) = U (B+S5;B+S5CA+S), 
BeA(G) 
e(A) = U (BB+SCA), 
BeA(G) 
K(A) = U (BB+ScCA+5S), 
BeA(G) 
a(A) = U (B+S;B+SC A). 
BeA(G) 
We now let 
f(B) = 


S, h( A) = A+ S in the first case; 
(A) = A in the second case; 

(A) = A+ S in the third case; and 
S, h(A) = A in the fourth case; 


while g(B) = B + S in all four cases. 


We can think of the points as atoms and the sets x + S as molecules. Then 6(A) and a(A) 
consists of molecules, the latter of those that are contained in A; whereas ¢(A) and «(A) consists 
of centers of molecules (which makes sense if the structuring set S is symmetric). 


( 
9.4 Lineally Convex Hartogs Domains 


Abstract of this sectionWe study lineally convex domains of a special type, viz. Hartogs domains, 
and prove that such sets can be characterized by local conditions if they are smoothly bounded. 


9.4.1 Introduction to the present section 


Lineal convexity is a kind of complex convexity intermediate between usual convexity and 
pseudoconvexity. More precisely, if A is a convex set which is either open or closed, then A is 
lineally convex (this is true also in the real category), and if © is a lineally convex open set in C”, 
the space of n complex variables, then 0 is pseudoconvex. Now pseudoconvexity is a local 
property in the sense that if any boundary point of an open set 0 has an open neighborhood w 
such that (21 w is pseudoconvex, then 0 is pseudoconvex; the analogous result holds for 
convexity. But it is well known that the property of lineal convexity is not a local property in this 
sense—for easy examples see SubSection 9.4.3. The purpose of this section is to investigate to 
what extent this is true for sets that are of a special form: the Hartogs domains. 
Let us now give the main definition. 


Definition 9.4.1 A set A in C” is said to be lineally concave if it is a union of hyperplanes. It is 
called lineally convex if its complement is lineally concave. 


A lineally convex set whose boundary is sufficiently smooth satisfies a differential condition. 
Let p be a defining function for 0 (see Definition 9.4.18), and let H and L denote, respectively, 
the Hessian and the Levi form at a boundary point a of 0. Then the differential condition says 
that 


|H(s)|<L/(s) for all vectors s € Tc(a), 
(9.41) 


where Tc(a) is the complex tangent space at the point a. See SubSection 9.4.5 for details. Every 
lineally convex domain of class C’ satisfies the differential condition—for the converse, see 
Section 9.6. Here we shall prove that this is so in the special case of Hartogs domains, which we 
now proceed to define. 


Definition 9.4.2 A Hartogs set in C" x C is a set which contains, along with a point 
(z,t) € C” x C, also every point (z,t’) with |t'| = |t|. It is said to be a complete Hartogs set if 
it contains, with (z,t), also (z,t') for allt’ with |t’|<|t]. 


Here we shall study open and complete Hartogs sets; they are always defined by a strict 
inequality |t| < R(z), thus 


2 = {(z,t) € C” x CG; |t] < R(z)}, 
(9.42) 


where R is a function on C” with values in R,. 


Given R, we define a set w in C” by w = {z € C"; 0 < R(z)< + oo}. We shall say that 0 is 
a Hartogs domain overy, or that « is the base of Q, if (9.42) holds with R(z) positive if and only 
if ze w. 

Most of our results will be concerned with the case n = 1, thus 


Q = {(z,t) Ee Cx C; |é| < R(z)} = {(z,t) ew x C; |t| < R(z)}. 
(9.43) 
The main result here is the following theorem. 


Theorem 9.4.3 Let Q be a bounded complete Hartogs domain in C” with boundary of class C?. 
If 0 satisfies the differential condition (9.41) at all boundary points, then Q 1s lineally convex. 


Thus for complete Hartogs domains, the property of being lineally convex is a local property. 
Next we consider sets with R of class C’ in w but which do not necessarily have a smooth 
boundary at points (z, t) with z € Ow. 

In this case we prove: 


Theorem 9.4.4 Let @ be a bounded open set in the complex plane C. If the closure of w is not a 
disk, then lineal convexity over @ is not a local condition: we can find a Hartogs domain Q over 


@ and two open sets wo and w, such that the Hartogs domains §2; over w, are lineally convex, 


j = 0,1, but their union 2Q = Qo U 2; Is not. If on the other hand q is a disk, and Q 1s a Hartogs 
domain satisfying the differential condition (9.41) at all boundary points over w, then Q is 
lineally convex. 


Corollary 9.4.5 Let w be an open set in C which is equal to the interior of its closure, and let Q 
be a Hartogs domain over w. Then the differential condition (9.41) imposed on all boundary 
points over @ is equivalent to lineal convexity if and only if w is a disk. 


9.4.2 Weak lineal convexity 

There are several other notions related to lineal convexity: 

Definition 9.4.6 An open connected set is called weakly lineally convex if through any boundary 
point there passes a complex hyperplane which does not intersect the set. An open set is said to 


be locally weakly lineally convex? if through every boundary point a € 022 there is a complex 
hyperplane Y passing through a such that a does not belong to the closure of Y 92. 


3There are actually two versions of this concept: see Definitions 9.5.7 and 9.5.8. 


It is not difficult to prove that local weak lineal convexity implies pseudoconvexity. 
For complete Hartogs sets it is very easy to see that weak lineal convexity implies lineal 
convexity: 


Lemma 9.4.7 A complete Hartogs domain which is weakly lineally convex and has a lineally 
convex base is lineally convex. 


Proof Let (z°,t°) € C” x C be an arbitrary point in the complement of 0, a Hartogs domain 
defined by (9.42). If R(z°) > 0, then the point (z°, R(z°)t°/|t°|) belongs to OQ, and if is 
weakly lineally convex, there is a hyperplane passing through that point which does not cut 0. 
Then the parallel plane through (2°, ¢°) does not cut © either. If R(z°)<0, then 2 does not 
belong to the base, and a hyperplane with equation ¢ - z = ¢- z° will do, since the base is lineally 
convex. This proves the lemma. 


9.4.3 The non-local character of lineal convexity 


The domain 
V= {(z,t) € C?, jt] < |z|} 
(9.44) 


is easily seen to be lineally convex. Indeed, if (zo, to) ¢ V with to # 0, then the complex line 
{(z,t); zot = toz} passes through (zo, to) and does not cut V; if on the other hand to = 0, we 
can for instance take the line {0} x C. A simple example of a domain that is locally lineally 
convex but not lineally convex can be built up from this set as follows. 


Example 9.4.8 (Kiselman 1996, Example 2.1.) 


FIGURE 9.1 


An open connected Hartogs set in C which is locally weakly lineally convex but not weakly lineally convex. Coordinates 
(z,t) € C?; (a, y, |t]) € R°. 


Define first 


{(z, t); |z| < land|t| < |z— 2]}; 
Q_ ={(z,t); |z| < land|t| < |z+2]}, 


and then 
Qo= 2.023 2 = {(z,t) € 2; |¢| <r}, 


where r is a constant with 2 <r < 1/5. All these sets are lineally convex. The two points 
(Hi, V5) belong to the boundary of {29; in the three-dimensional space of the variables 
(Re z, Im z, |t|), the set representing 29 has two peaks, which have been truncated in {2}. 

We now define 92" by glueing together Qo and (2): Define 92" as the subset of Qo such that 
(z,t) € Q5 ifIm z > 0; we truncate only one of the peaks of 2. 

The point (i—¢,r) for a small positive ¢ belongs to the boundary of 92" and the tangent 
plane at that point has the equation t = r and so must cut QQ" at the point (—i+ e,r). Therefore 
QQ" is not lineally convex, but it agrees with the lineally convex sets Qo and 2Q) when Im z < 6 
and Im z > —64, respectively, for a small positive 6. The set has Lipschitz boundary; in particular 
it is equal to the interior of its closure. 


Proposition 9.4.9 Let wo and w; be two bounded open subsets in the complex plane such 

that none is contained in the closure of the other. Then there exists a Hartogs domain over 
W = Wo Uw) that is not lineally convex, but is such that the subsets 2; over w; are both lineally 
convex, 7 = 0,1. 


Proof Take two points a € w;\Wo and b € wo\W,, which exist by hypothesis. It is no restriction 
to assume that a = i, b = —i. Then take c > 0 so large that w is contained in the disk of radius 
c — 1 and with center at the origin. We then define as in Example 9.4.8, 


Qo= {(21) EC’; 


i| <|zte| and |t| < |2tiat Ve +1)]}, 
and 
2, = {(z,t) € 2; |t| <r}, 


where ris a number slightly smaller than Vc? + 1 but so close to that number that the peak that 
we have truncated in §2; near i € C lies outside wp, and the peak near —i lies outside w,. This is 
possible since we have assumed that i € Wo, —i € W,, and Mp and $2, differ only above small 
neighborhoods of i and —i. These neighborhoods shrink to {i, —i} as r increases to Vc? +1. We 
now define Q to agree with (2; over w; j = 0,1. The conclusion is as in Example 9.4.8. 


9.4.4 Smooth vs. Lipschitz boundaries 


The lineally convex set (29 constructed in Example 9.4.8 has the remarkable property that it 
cannot be approximated by lineally convex sets with smooth boundary. Its boundary, which is 


Lipschitz, cannot in any reasonable way be rounded off if we want to preserve lineal convexity. 
This is why we shall continue this investigation to see whether smoothly bounded sets admit a 
passage from the local to the global. 

Before doing so, however, we shall illustrate the difference between domains which can be 
approximated by smoothly bounded lineally convex domains and those that have only Lipschitz 
boundary. 

Let © be a complete Hartogs domain with R a function of class C!, w being the open set 
where R > 0. Often it will be convenient to use not R but h = R? to define the set, thus 


Q = {(z,t) Ew x C; |t| < R(z)} = {(z,t) €w x C; |t/? < h(z)}. 


(9.45) 
The complex tangent plane at a boundary point (zo, to) with zo € w has the equation 
t —tpo = a(z— 20), where a = see) = ae ; 
(9.46) 
The tangent plane intersects the plane t = 0 in the point 
b(z0) = zo — at = 20 — mo : 
(9.47) 


If R.(zo) = 0, the tangent plane has the equation t = to, and in this case we define b(zo) = 00, 
the infinite point on the Riemann sphere S* = C U {oo}. 


Proposition 9.4.10 Let R € C1(C) and define Q by (9.43). If Q is bounded and lineally convex, 
then b(z), defined by (9.47), does not belong to w, so that b is a continuous mapping from @ into 
S?\w. Its range contains S?\B. 


Proof Clearly b is continuous as a mapping into C except where R, = 0. Near such points, 
however, 1/6 is continuous. The point (b(zo), 0) cannot belong to ( since () is lineally convex; 
thus b(zo) ¢ w. From every point (z, 0) outside the closure of 0 we can draw a tangent to : this 
shows that the range of b contains C\W; clearly it also contains ~. 


Corollary 9.4.11 If 0 is as in Proposition 9.4.10, then Q is connected. The same is true if Q is the 
union of an increasing family of bounded lineally convex sets §2; defined by functions 
R; € C*(C). 


Proof Let w, be a component of @ and let 2; be the set over w,. Then the image of the boundary 


of 2, under b contains S?\@z. Since b(z9) ¢ w there can be no other component: we must have 
w, = w. The statement about U.Q; is now immediate. 


Corollary 9.4.11 should be compared with the following easy result for Lip-schitz boundaries. 


Proposition 9.4.12 Let be any open set in C. Then there exists a Lipschitz continuous function 
R€C(C) such that w is the set where R is positive and the set Q defined by R is lineally 
convex. 


Proof We define R(z) = i E — a}. The set is lineally convex since it is an intersection of sets 
agw 


of the type V defined in (9.44). 


The set Msup Rr where the function R assumes its maximum can be rather arbitrary as shown be 
the next proposition. 


Proposition 9.4.13 Given any closed set M in the complex plane such that its complement is a 
union of open disks of radius r there exists a Lipschitz continuous function R such that 
Msup Rk = M and the domain 2 defined by (9.45) with this R is lineally convex. 


z—a 


Proof Define R(z) = min (+ inf ) where A is the set of all centers of disks of radius r 
ace 


in the complement of M. 
But when R is of class C!, the set Mogup R iS convex: 
Theorem 9.4.14 Let R: C" > R be a function of class C' or more generally a continuous 


function which is the limit of an increasing sequence of functions R, of class C 1 in the sets 


{z; R;(z) > 0}. We assume that R is positive only in a bounded subset of the complex plane. 
The functions R; define open sets §2;, which we assume to be lineally convex. Then the set 


Mink = {2 R(z) = supR(w)} is convex. 


This could be proved here, but it is more easily done with the methods of Section 9.7: see 
Theorems 9.7.29 and 9.7.30. 

If a set does not have a boundary of class C', we cannot give a meaning to the notion of 
tangent plane. However, if the set is the union of an increasing family of sets with smooth 
boundaries, it is possible to use instead their tangent planes and then pass to the limit. Such limits 
of tangent planes can serve as well, as explained in the following lemma. 


Lemma 9.4.15 Let Q be the union of an increasing family of open lineally convex sets 22; with 
boundaries of class C’. Let (jx) be a sequence tending to +00, and let Y, be the complex 
tangent plane of 092;, at some point in the boundary of 22;,, k € N. Assume that the Y; 
converge to a hyperplane Y in the topology of hyperplanes. Then Y does not intersect (2. 


Proof Suppose there is a point z € Y 1 2. Then also z € Y 1 9Q;, for all large k. Since 2;, is 
open, there is a ball B<(z,e) C 2;, for large k, say for k>ko. But then Y; intersects B<(z, €) 


for all large k, say for k>k,. Thus Y;,M 2;, is non-empty for all k> max(ko, ki), contradicting 
the lineal convexity of 2;,. 


To recognize such limits of tangent planes we shall use the concept in the following definition. 


Definition 9.4.16 Let X be any subset of C” and a a point in the boundary OX. We shall say that 
a complex hyperplane Y is an admissible tangent plane to OX at a if there exists an open set A 
with boundary of class C' such that A and X are disjoint, a belongs to the boundary of A, and Y 
is the complex tangent plane to A at a. 


Proposition 9.4.17. Let 22 C C” be the union of an increasing family of lineally convex open 
sets 2; with boundaries of class C!. Then any admissible tangent plane Y to 092 is the limit of a 
sequence of tangent planes Y; to 022;. Therefore, in view of Lemma 9.4.15, Y cannot intersect Q. 


Proof Let a and A be as in Definition 9.4.16. By a coordinate change we may suppose that a = 0, 
that the real tangent plane to OA at the origin has the equation y,, = 0, and that A is defined by 
an inequality y, > f(z1,..-,2n—1,€n) near the origin for some function f of class C!, which 
consequently vanishes at the origin together with its gradient. Write 2’ = (z1,...,2n-1) € C™?. 
We then know that all points in QQ _ satisfy yn,<f(z',an). Define 
g(2’,2n) = f(z’, @n) + |l2"||2 +22, and let A, be the set of all points such that 
Yn > g(z', Ln) — c. We let c = c; be the largest real number such that A, and Q; are disjoint. 
Now 0 € 02 and 2; 7 22; therefore we can be sure that c; tends to zero as 7 —> oo. There is a 


point z which is common to the boundaries of A,, and 2;. Since A and 92; are disjoint, we have 
, oe Pgs é 
|| (2!) \|3 + (z,) <c;. The real tangent plane to OA,, at # is identical to the real tangent plane to 


O02; at that point. We can control its slope, for the gradient of gis 
gradg = gradf + grad{(\|2'||; + <7), 


which is continuous and vanishes at the origin. Since ((z’)', 2.) tends to the origin, this shows 
that the real tangent plane to 0A,, at z must be close to the real hyperplane y,, = 0 if jis large, 
and then the complex tangent plane to 0A,, at # is close to the complex hyperplane z, = 0. The 
last statement now follows from Lemma 9.4.15. 


9.4.5 Differential conditions 


Definition 9.4.18 Let Q be an open set in C" with boundary of class C’. Then a function 
p€C'(C”) is called a defining function for 0, if dg #0 wherever p=O0 and if 
2 = {z € ©”; p(z) < 0}. (Here the differential d is defined in (9.2).) 


Definition 9.4.19 The complex tangent space at a point a on the boundary of Q is defined by 


and will be denoted by Tp(a). The complex tangent plane is then a + Tc(a); it is contained in 
the real tangent plane a + Tg (a). 


To be able to characterize sets by infinitesimal conditions, we shall describe boundaries and 
their curvature using defining functions and the Hesse and Levi forms. We now give the needed 
definitions. 


Definition 9.4.20 The complex Hessian (or complex Hesse form) of a function f of class C? is 
defined to be 


02 
CoS»). 5a (e)tste zeEc", tec’, 


Zj0Z 
(9.48) 
a quadratic form in the t. 
Definition 9.4.21 The Levi form of f is the Hermitian form 
L(t) = >> o"f (z)t;, 2EC", tec. 
0z;0Z,, 
(9.49) 
We say that 0 satisfies the Levi condition ata € O if 
L,(a;t)>0 whent € Tc(a), 
(9.50) 


where p is a defining function for Q; and that 0 satisfies the strong Levi condition at a if strict 
inequality holds in (9.50) fort # 0. 


Definition 9.4.22 The real Hessian of a function f of real variables x1,..., Lm 18 
HF (2; s) = eon (x) 858%, xER™, sER”, 


(9.51) 
a quadratic form. 


When a function of n complex variables is given, its real Hessian in the 2n real variables 


(Re z1, Im z1,..., Re zn, Imz,) can be expressed using its complex Hessian and its Levi form 
as 
HH} (z;s) = 2(Re HP (z;t) + Ly(st)), 


forzE C",s € R*, te Cr”, tj = Soj-1 + 189;. 

Thus the characterization of convexity mentioned in the introduction is_ that 
Re HY (a; t) + L,(a;t) be nonnegative for all a € 02 and all t € Tr(a). For a lineally convex 
set the same inequality holds for all t € T¢(a). It is then equivalent to L(a;t)>|H(a;t)| for 
a € ONandt € Tc(a). 


Definition 9.4.23 We shall say that a set Q with boundary of class C? satisfies the Behnke- 
Peschl differential condition at a boundary point a of Q if 


JHE (a; s)|<L,(a; s) forallvectors s € Tc(a), 
(9.52) 


where p is a defining function for Q. We shall say that Q satisfies the strict Behnke—Peschl 
differential condition at a if there exists a positive number ¢ such that we have 


|S (a; 8) |<Lp(a; 8) — ells|l3 


(9.53) 


for alls € Tc(a). 


It is easy to prove that these conditions are invariant under complex affine mappings. They also 
do not depend on the choice of defining function. They were introduced for n = 2 by Behnke 
and Peschl (1935:169). 

These conditions should be compared with the differential condition for convexity: 
|H,(s)|<L,(s) for all vectors s in the real tangent space Tp(a). This is a local condition, and it 
is well known that it is equivalent to convexity of 0. The proof of this fact most conveniently 
goes via approximation of the set by sets satisfying the corresponding strong condition, i.e., 
|H_,(s)|<L,(s) — e||s||3 for a positive ¢ and for all s € Tr(a). 

The following two lemmas are well known; cf. (Zinov ev 1971) and (Hérmander 1994: 
Corollary 4.6.5). We include them for ease of reference. 


Lemma 9.4.24 Let Q be an open subset of C” with boundary of class C?. If Q is locally weakly 
lineally convex, then Q satisfies the Behnke-Peschl differential condition at every boundary 
point. 


Proof Let a be an arbitrary boundary point of a locally weakly lineally convex open set 0. Then 
there exists a complex hyperplane through a that does not cut 0 close to a. This hyperplane 
cannot be anything but Tc(a) since the boundary is of class C!. Therefore if we take an arbitrary 
vector s € Tc(a) and consider the function y(t) = p(a+ ts) of a real variable t, its second 
derivative must be non-negative at the origin. If we express the condition y”(0)>0 in terms of H 
and L we get Re H(s) + L(s)>0, which, since His quadratic and L sesquilinear, is equivalent to 
|H|<L. 


Lemma 9.4.25. Let Q be an open subset of C” with boundary of class C. If Q satisfies the strict 
Behnke-Peschl differential condition at every boundary point, then Q is locally weakly lineally 
convex. 


Proof With ¢ as in the proof of the previous lemma we must have y"’(0) > 0 if Q satisfies the 
strict Behnke—Peschl differential condition. This imples that T¢(a) cannot cut 0 close to a. 


It is known that if © is a connected open set with boundary of class C! which is locally weakly 
lineally convex, then Q is weakly lineally convex; see, e.g., (H6rmander 1994: Proposition 4.6.4). 
We shall come back to this result in SubSection 9.4.7. 


9.4.6 Differential conditions for Hartogs domains 


In this subsection we shall see what the differential conditions look like in the case of a complete 
Hartogs domain in C*. Let 0 be a complete Hartogs domain in C? defined by (9.45). If his of 
class C', we can choose as its defining function 


p(z,t) = tt — h(z), (z,t)ECxC. 


It must satisfy d’p 4 0 when p = 0, which means that d'p = tdt — h, dz 4 0 when |¢|? = A(z). 
Since the first term of d’p is tdt, which is non-zero everywhere except in the plane t = 0, the 
only condition is that h, 4 0 when h = 0, i.e., that A itself shall be a defining function in C. It 
defines a subset w of the complex plane over which (0 is situated. 


Lemma 9.4.26 Let h be a defining function of an open set w in C of class Ck, k>1. Then the 
complete Hartogs domain in C? defined by (9.45) has boundary of class Ck. When k>2, it 
satisfies the Behnke—Peschl differential condition at every boundary point if and only if h 
satisfies the condition 


h,|? 
| a Zligg + 


h.z| wherever h > 0. 


(9.54) 


Furthermore 2 satisfies the strict Behnke—Peschl differential condition if and only if there is strict 
inequality in (9.54). 


Proof Let us look at the Hessian and Levi forms of p(z,t) = |t|? — h(z). They are, respectively, 
H(s) = —h,,s? and L(s)=-h,g|s,|’+|s,|°,.  s=(s1,82) € C?. 
The differential condition |H|<L takes the form 
|Aze||s1|?< — hez|sil? + |s2|? for all s € To(a). 


The tangent plane is defined by —h,s1 + ts2 = 0. When t £ 0 we use this equation to eliminate 
Sy: the condition takes the form (9.54). Near t = 0 we eliminate instead s, and get 


(hiz at: hzz ) ae <1. 


This inequality is satisfied, even strictly, at all boundary points sufficiently close to t = 0, 
provided h, ~ 0 near h = 0. Therefore, if A is a defining function for @, then p is a defining 
function for Q and condition (9.54) implies the Behnke—Peschl differential condition at all 
boundary points of ©, including those where t = 0. Conversely, if p is a defining function for 0, 


then his a defining function for w, and the Behnke—Peschl differential condition for p implies the 
condition (9.54) for A. 


Remark 9.4.27 We can also express the Behnke—Peschl differential condition (9.54) in terms of 
the radius R = Vh. It becomes 


(9.55) 


which is less convenient to work with than (9.54). If h is concave, then h,z + |h,,|<0, so that 
(9.54) holds. More generally, if R is concave, then R,z+|Rzz|<0, which implies that (9.55)) 
holds. It is also possible to express the Behnke—Peschl differential condition in terms of the 
function f = — log R. It then takes the form 


(9.56) 


We note that (z,t) ++ |t|? —h(z) is convex if and only if h is concave, and that 
(z, t) + log ||z||2 + f(z) is plurisubharmonic if and only if f is. 


9.4.7 Approximating bounded lineally convex Hartogs domains by smoothly bounded 
ones 


Theorem 9.4.28. Let 
2 = {(z,t) € C?; |t|? < h(z)} 
(9.57) 


be a bounded complete Hartogs domain in C? with boundary of class C?. Suppose 0 satisfies the 
Behnke-Peschl differential condition at all boundary points. Then Q can be approximated from 
the inside by Hartogs domains 


2. = {(z,t); |t] < Re(z)} 


which satisfy the strict Behnke—Peschl differential condition all boundary points (z,t) except 
those where h,(z) = 0. In fact, we can take hz = h — € withe positive and small enough. 


Proof This is an instance where it is more convenient to use / rather than R. The Behnke—Peschl 
differential condition (9.54) contains the value of f only at one place, and hz; = h —« has the 
same derivatives as h, So we can write 


|r|? 
h—-e > h Zhi + 


less 


except when h, = 0. Thus the boundary of 2, satisfies the strict Behnke—Peschl differential 
condition except at the points where h, = 0. So far the argument is valid for all positive ¢. We 
need to check that h, is a defining function; otherwise we cannot apply Lemma 9.4.26. But the 
gradient of h, is the same as that of h, which is non-zero when h = 0, hence also when h, = 0, 
provided ¢ is small enough. Thus A, is indeed a defining function for all small ¢, proving the 
theorem. 


We shall now see that the approximating sets (2, that we constructed in Theorem 9.4.28 are, in 
fact, lineally convex. Let us agree to say that a complex plane with the equation z = constant is 
vertical and a plane with the equation t = constant is horizontal. 


Proposition 9.4.29 Let Q be a bounded complete Hartogs domain in C? with boundary of class 
C® satisfying the strict Behnke—Peschl differential condition except possibly at the points where 
the tangent plane is horizontal. Then Q is lineally convex. 


We shall need the following three lemmas. 


Lemma 9.4.30 Let (Q be as in Proposition 9.4.29 and let L be a complex line in C* which is not 
horizontal. Then L \ QQ consists of a finite number of open sets bounded by C? curves obtained 
as transversal intersections of L and 092 (and LM O22 consists of these curves plus a finite 
number of isolated points). 


Proof Take an arbitrary boundary point a and let L be a complex line through a which is not 
horizontal. If L is the tangent plane, L = a + Tc(a), then the proof of Lemma 9.4.25 shows that 
L intersects 2 near a only in the point a. If, on the other hand, L is not the tangent plane, then 
LN (a+Tc(a)) 4 L, so OP cuts L transversally, and 022M L is a C curve in L near a. Thus 
L © 02 consists of a number of C’ curves plus isolated points—by compactness there can only 
be finitely many curves and points. 


Lemma 9.4.31. Let Q and L satisfy the hypotheses of the previous lemma. Then 221 L is 
connected, and 2 (a + Tc(a)) is empty for alla € O22. 


Proof We shall follow the proof of Proposition 4.6.4 in (Hérmander 1994)—-we only have to be 
careful to avoid horizontal planes. Let (z;,t;), 7 = 0,1, be two points in LM 2. We have to 
prove that they belong to the same component of LM 92. Suppose first that both f and t, are non- 
zero. Since Q is connected, there is a curve y which goes from (0) = (zo, to) to y(1) = (21, #1). 
We can actually do this in such a way that the complex line L, that contains 7(0) and ¥(s), 
0 < s<1, is never horizontal. Indeed, we first go from (zo,to) to (zp,0) along a curve in the 
plane z = zo avoiding (zo, t1); then along a curve in the plane t = 0 from (29,0) to (z1, 0); and 


then finally from (21, 0) to (z1, t1) along a curve in the plane z = z, avoiding (21, to). (We know 
that to ~ t,.) Thus none of the lines L, is horizontal, and we can apply Lemma 9.4.30 to them. 
Consider the set C of all s €]0,1] such that y(0) and y(s) belong to the same component of 
L, 9. Then certainly C contains all sufficiently small numbers, for (0) and +(s) are then in 
the line z = zo, whose intersection with Q is a disk. The set Cis open as a subset of ]0, 1] in view 
of Lemma 9.4.30, but so is its complement with respect to ]0, 1]. Since it is non-empty, it must 
contain 1, i.e., (zo, to) and (z;,t1) belong to the same component of LM 92. If one of 6, t; is 
zero, we choose a point with non-zero second coordinate in the neighborhood and argue as 
above. 

Consider now a tangent plane L = a+ To(a) and planes L; = a, + Tc(a) parallel to it, 
where we write ag = (Zo, (1 — €)to) if a = (Zo, to). We already know from Lemma 9.4.25 that L 
cannot intersect 0 close to a. However, it cannot cut Q, at all, for if it did, then a parallel plane L, 
for some small positive ¢ would intersect 0 in a component close to a and another nonempty set 
at some distance from a, thus in a disconnected set. This proves Lemma 9.4.31. 


Lemma 9.4.32 Let Q be as in Proposition 9.4.29 and leta € 092 be such that the tangent plane is 
horizontal. Then 2.) (a+Te(a)) is empty; in other words R has a global maximum at a. 
Consequently any horizontal plane L intersects Q in finitely many open sets bounded by C? 
curves obtained as transversal intersections of L by 022. 


Proof Let (zo, to) be a boundary point such that the tangent plane is horizontal, i.e., Rz(zo) = 0. 
Suppose the tangent plane cuts 0 in some point (z1, t1). We must then have t; = to. Since 0 and 
its base w are connected, we can find a curve y in w connecting Zp to z,, say y(s) = 2s, 8 € [0,1] 


Consider now the tangent planes at the points (z;,,R(zs)); we denote them by 

L, = (2s, R(zs)) + To(zs, R(z5)). It is no restriction to assume to > 0, so that R(zo) = to. We 
know that Lp is horizontal, but certainly not all the L, can be horizontal, since 
R(z1) > |t1| = |to| = R(zo). Let so be the infimum of all s such that L, is not horizontal; we 
must have 0<so < 1. The planes L, with O<s<sSpo are identical and all intersect 0 in the point 
(z1,t1). It is now clear that there exists a tangent plane L, with s just a little bit larger than sp 
which is not horizontal and still cuts ©. This contradicts Lemma 9.4.31. 
Proof of Proposition 9.4.29 We know from Lemma 9.4.31 that a tangent plane which is not 
horizontal does not intersect 0; we obtain the same conclusion from Lemma 9.4.32 for a 
horizontal tangent plane. Thus © is weakly lineally convex. Lemma 9.4.7 shows that this implies 
lineal convexity. 


We can now finally state the main result of this section: 


Theorem 9.4.33 Let Q be a bounded complete Hartogs domain in C? with boundary of class C?. 
If Q satisfies the Behnke—Peschl differential condition (9.52) at all boundary points, then Q is 
lineally convex. 


Proof Using Theorem 9.4.28 we construct open sets 2;, which tend to Q. Also, if R(zo) > 0, the 


tangent plane of 022, at (<, (20)? — :) tends to that of 0.2 at (zo, R(zo)). The sets Q, are 


lineally convex by Proposition 9.4.29. Then also their limit © is lineally convex. Indeed, if a 
tangent plane to 02 intersected Q, then it would cut also (2, for all sufficiently small ¢, and then 
also for ¢ small enough the corresponding tangent plane to 092. would cut ,. This is a 
contradiction. 


9.4.8 The non-local character of lineal convexity, revisited 


Having settled the question of lineal convexity of smoothly bounded Hartogs domains we now 
turn to sets of the form 


Q = {(z,t) Ew x C; |t| < R(z)} = {(z,t) €w x C; |t/? < h(z)}, 
(9.58) 


where « is a given open set in C and his a C’ function in the closure of @ satisfying h > 0 and 
the Behnke—Peschl differential condition (9.54). Its boundary is smooth enough over points in @, 
but is only Lipschitz at points over Ow. It turns out that when @ is a disk, then the Behnke—Peschl 
differential condition implies lineal convexity: we shall study this question in SubSection 9.4.9. 
On the other hand, if w is a set such that @ is not a disk, then the Behnke—Peschl differential 
condition does not imply lineal convexity. This is the topic of the present subsection. 

The property of being a disk is invariant under Mobius mappings, and disks are the only sets 
that remain convex under all Mébius mappings. This is a kind of explanation for the phenomenon 
we encounter here, and it is therefore natural to study how the Behnke—Peschl differential 
condition (9.52) behaves under Mobius mappings. This is explained in the next lemma. 


Lemma 9.4.34 Let Q be a Hartogs domain in C? defined by |t| < R(z), let a,b,c,d be four 
complex numbers with ad—bc#0, and let 92; be the Hartogs domain defined by 
jt] < Ri(z) = |c+ dz|R((a + bz)/(c+dz)). Then Q and 2, are lineally convex 
simultaneously. The two functions h and hi(z) = |c + dz|*h((a + bz)/(c+dz)) satisfy the 
Behnke-Peschl differential condition (9.54) simultaneously. 


Proof Take constants a, 6 and c of which not both of a and # are zero, and consider the mapping 
(C\{0}) x C x C3(z9, 21, t) 4 (21/20, t/z) € C?. 


Under it the pull-back of the hyperplane of equation c+ az-+ (St =0 is the hyperplane of 
equation czp + az; + Bt = 0. It follows that the pull-back of a lineally convex set in C? is a 
complex homogeneous lineally convex set in C’. Now any linear mapping of the form 


C23(z0, 21, t) > (czo + dz, az + b21,t) € C3 


with ad — bc 0 preserves lineal convexity, and mappings 


C?3(z0, 21, t) > (1 Genes t ) eC 


) czgtdz,? czot+dz 


preserve lineally convex sets which are complex homogeneous. If we transport this back to C* 
we get a mapping of the form 


(z, t) (ae ae) 


This proves that Q and 2; as defined in the statement of the lemma are lineally convex at the 
same time. The statement about the differential condition for A and h, can be verified directly, 
perhaps most easily if we check it for the special mappings z+> c+ dz and z+> 1/z, which 
together generate all Mébius mappings. 


Lemma 9.4.35 Let K be a compact subset of C with connected complement. Assume that K is 
not a disk. Then there exists a closed disk D, containing K such that K 1 0D, has at least two 
components. 


Proof Let Dp be the closed disk of minimal radius that contains K. By hypothesis K # Do and 
C\K is connected, so there exists a point a9 € ODo\K. Let H be an open half plane that 
contains K but is such that aj ¢ H. Now consider the closed disk D, of minimal radius among 
those that contain K and have OH as a tangent. We claim that there are four points 
a,b,c,d € OD, which are in that order along the circumference and with a,c ¢ K, b,de€ K. 
This will show that b and d belong to different components of K M OD. To find these points we 
argue as follows. Let a be the point of OD, at which OH is tangent; thus a € OD; anda ¢ K. 
Next, Dj 2. Dp, so there is a point c € OD,\Do. Thus c ¢ K. Finally we claim that there are 
two points b,d € 0D, K on either side of the segment /a,c/. This is so because if one of the 
arcs from a to c were disjoint from K, then it can easily be seen that D, would not be minimal 
among the disks that contain K and are tangent to OH. This completes the proof. 


Theorem 9.4.36 Let w be a bounded connected open subset of C such that the complement S?\G 
of its closure with respect to the Riemann sphere S* = C U {oo} has at least one component 
which is not a disk. Then there exists a Hartogs domain defined by a smooth function and with 
base @ such that it is not lineally convex, although w = wo U w, and the Hartogs domains over 
wg and wy are both lineally convex. In particular the function defining Q satisfies the Behnke- 
Peschl differential condition (9.54). 


Proof Let K be the complement of a component of $?\@ which is not a disk; thus K contains 0. 
Moreover the complement of K is connected and 0K C Ow. We may assume that K is compact: 
if not we use a MGbius mapping to reduce ourselves to that case. Let a, b, c,d € OD, be the four 
points whose existence is guaranteed by Lemma 9.4.35; recall that b,d € K anda,c ¢ kK. Now 
take a new closed disk D, which does not contain a,b, or d, but contains c in its interior, and is so 
close to D, that b and d belong to different components of K\ D2. This is possible because a 
does not belong to K. Now we map Dj onto the closed right half plane, taking a to O and some 


point outside K and near c to infinity. We are thus reduced to a situation where K is still compact 
in C, whereas 0Dz is the imaginary axis, with a = 0 and Imb and Imd of different signs, say for 
definiteness Im b < 0 and Imd > 0. Moreover we can take D, so close to D, that the points in K 


which are not in Dy, are never real. Then we can define a function R as follows. First take a 


smooth concave function y of a real variable such that w(s) = 1 when s>0 and 7#(s) < 1 for 
s < 0, but still so that ~(Re z) > 0 for all points z € &. Then define 


R(z) w(Rez) whenz€w, Rez <0, Imz <0; 
Li = 
1 at other points in w. 


This function is continuous, even identically one, in a neighborhood of the intersection of @ and 
the real axis. 

The tangent plane at a point (zo, to) € OM with zp € w has the equation (9.46). In particular, 
we may take to = R(zo) and get 


t = R(z) + 2R,(z0)(z — 20). 


In the present case R is locally a function of Rez, say R(z) = k(x), so that R, = k,/2 is real. 
Thus the tangent plane is 


t = R(z0) + kx (ao)(2 — 20) = R(20) + ka(xo)(@ — 20) + ika(x0)(y — Yo); 


and, writing z = Zp + 21, we obtain 
|t|? = R(z0)” + 2ke(ao)R(z0)21 + ke(x0)"2? + ke(x0)°y?. 
When x; < 0 and k,(x0) is positive and small, 
jel? ~ R(zo)? + 2ke(ao)R(z20)21 < R(z0)’. 


Since @ is connected and has the point b on its boundary, we can choose Zp such that yo < 0 and 
zo <0 with k,(ao) arbitrarily small, so small that indeed |t| < R(zo). Then we choose 
Z= 2%) +2, € w with Imz > 0. Thus R(z) = 1, so the tangent plane at (z, R(zo)) cuts 0 ina 
point above z. This proves that Q is not lineally convex. However, if we look at the parts of w 
where Imz > —e and Imz < é respectively, then R is the restriction of a globally concave 
function in each of them and therefore defines a lineally convex set. 


Theorem 9.4.37 Let w be a bounded open set in C such that S*\@ is not connected. Then there is 
a function h € C®(®), h > 0, which satisfies the Behnke—Peschl differential condition (9.54) 
but is such that the Hartogs domain it defines over o is not lineally convex. 


Proof If one of the components of S?\@ is not a disk, we already know the result by Theorem 
9.4.36. The case when all components of S$”\@ are disks remains to be considered. This means 
that @ is a disk from which countably many disks (at least one) have been removed. Any one of 
these holes can be moved by a Mobius transformation so that it becomes concentric with the 
outer circumference of @; in other words @ is an annulus r9<|z|<r; from which possibly a 
number of disks have been removed. It is clearly enough to consider the case of the annulus, for 
the possible presence of other holes will not destroy our conclusion. 

So assume @ is the annulus ro<|z|<r1 and define Ro(z) = 1 — ax? — by”, whereO <a<b 
and b is so small that Rp > 0 in &. Next define gy to be a concave C’™ function of one real 


variable such that y(s) = s for all s<1 — br? + € and y(s) = c when s>1 — ar2 — « for some 
positive ¢ and a suitable constant c; by necessity we must have c<1— are. Define 
Ri(z) = y(Ro(z)). We observe that Ro = R; in a neighborhood of the intersection of the 
imaginary axis and W. Both Rp and R, are concave in C, so the corresponding Hartogs domains 
over |z| <r, are convex and therefore lineally convex. It follows that the Hartogs domains over 
@ are lineally convex. Now define R to agree with Rp in the right half plane and with R, in the 
left half plane. Note that R(z) = Ri(z) =c at points z € w close to —ro, so that the tangent 
plane at a boundary point over such a point has the equation t = tg with |to| =e<1- are. But 
over a point z in @ close to m we have R(z) = Ro(z) > ¢, so the tangent plane t = to cuts . 
This proves that 0 cannot be lineally convex. 


9.4.9 Hartogs domains over a disk 


The Behnke—Peschl differential condition over a disk remains to be studied. We shall see that it is 
then equivalent to lineal convexity. 

We shall write D(c,r) for the open disk in the complex plane with center c and radius r, and 
just D for the open unit disk D(0, 1). 


Proposition 9.4.38 Leth € C?(D), h > 0, be a real-valued function which satisfies the Behnke- 
Peschl differential condition 


; 2 <1. 
(9.59) 


Let p € C?(R) be real-valued, decreasing and satisfy p<1 everywhere and y" <0 wherever 
y <1. Assume that there are constants a and A such that 


Re ane <a<l 


(9.60) 


and 


2zhz(z) 
h(z) 


<A < +00 


(9.61) 


wherever 0 < p(zZ) < 1. Then g(z) = y(zZ)h(z) satisfies the differential condition wherever 
(zz) > 0 and |z| < 1, provided y'/y" is small enough, more precisely if either A<1 or else 


els) ¢ 2(1-a) 


sora) S az When sissuchthat 0 < y(s) <1. 


Proof With g(z) = y(zZ)h(z) we have 


Gz = +2Zh+ phy, 
92 = yp" . oh +r 2p'Zh, + lize, 
Gz =": |2\?h+y'h + 2p'Re zh, + phiz. 


Thus what we have to prove is, writing rfor |z|, 


I>; 2 
eee Sr*p"h+ yh + 20'Re zh, + pha + p"27h + 2y'Zhz + phiz 


We expand the left-hand side and find that the term 2y’ Re zh, appears on both sides. We shall 
therefore prove 


Ee 92 SE ol holt + Oh lp"2"h + 29'Zhz + phx 


This formula follows from |h,|?/h>h,z + |h,,|, which holds by hypothesis, and 


? 


Pe Srp"h + o'ht lp" 2h + 2! Zh, 


(9.62) 


which we shall prove now. We divide both sides of this inequality by the positive quantity 
—r?"h (if g' is zero there is nothing to prove), and find the equivalent inequality 


! ~2 Is ! ' 
z Zhz 2zh, 
2 1 io | 2 2 - a 1 ig 1 iol h . 


Since —(y’)”/yy" is positive, it suffices to prove that 


2zh,(z) 
h(z) * 


ne a 
1+ t>|1 + tw when t = pM) and w = 


This inequality, in turn, follows from 


(1+ ¢)’>|1 + tw|? =1+4 2tRew + ¢?|w)?, 


which holds as soon as 2 + t>2Rew + t|w|?. By hypothesis Rew<a < 1 and |w|<A, so (9.62) 
follows as soon as either A<1 or else A>1 and t<2(1—a)/(A* —1). This proves the 
proposition. 


Example 9.4.39 As an example of the function g in Proposition 9.4.38 we let s9 be an arbitrary 
number such that 0 < s9 < 1 and take a smooth function 9 satisfying p(s) =1 for s<s9 and 
whose derivative is y'(s) = —C exp(—1/(s — s0)) for s > 89. Then we determine C to make 
(1) = 0; this means that we choose C to satisfy 


i 
al e V/s) dg — 1. 


0 


We note that y'(s)/sp"(s) = (s — s0)*/s, which varies between 0 and (1— 89)”. Thus if 
1 — 89 is small enough, we can conclude that the new function ~(zZ)h(z) satisfies the Behnke- 
Peschl differential condition (9.59) over the open unit disk and it agrees with h when |z|<./s0. 


We need to study condition (9.60) more closely. In fact it has a simple geometric meaning. 
Definition 9.4.40 Let a complete Hartogs domain 
2 = {(z,t) €w x C; |t|? < h(z)} 


be defined over a bounded domain w in C by a function h € C!(w), h > 0. Denote by (b(z), 0) 
the point at which the tangent at a point (z,t) € 022 with z € w intersects the plane t = 0 (put 
b(z) = oo if there is no such point in C). We shall say that Q satisfies the tangent condition if 


infd(b(z),w) > 0, 


ZEW 


(9.63) 


where d denotes the distance from a point to a set. 


If © is defined by a function h>c > 0 and is lineally convex, then it must satisfy the tangent 
condition, but not only that—we can deduce important quantitative information from its lineal 
convexity: 


Lemma 9.4.41 Let R € C1(w) be such that the set Q defined by (9.58) is lineally convex. Then 


infR infh 
om 
sup 


infd(b(z), w)> : 
ZEW hz 


R, 


2sup 
(9.64) 


If R>c > 0 ina, then O satisfies the tangent condition (9.63). 


Proof The tangent plane at a point (zo, t 9) € O02 with zy € w is given by equation (9.46), and 
b(z) is given by equation (9.47). The equation for the tangent can also be written as 
t = a(z — b(z0)). If is lineally convex, then this tangent cannot intersect 0, so we must have 
|t|>R(z) whenever z, zo € w. Thus 


jt] = ja(z — b(z0))|>R(z) for all z, zo € w; 


inserting the value of la = 2|R.(zo)| — hz(20)|//h(20) we obtain 


PAR e)) ~~ Thelzo)l 


z— (2)| RG) _ Vh()hG0) 


We now let z, Zo vary in @ to get the desired conclusion. 


The idea is to prove that the tangent condition is not only necessary as in Lemma 9.4.41, but also 
sufficient if @ is a disk, which we shall do in Proposition 9.4.42. We then proceed to prove that 0 
does satisfy the tangent condition under our hypotheses if @ is a disk. 


Proposition 9.4.42 Assume that h € C? (D), h > 0, satisfies the Behnke—-Peschl differential 


condition (9.59) and that Q satisfies the tangent condition. Let @ be the function constructed in 
Example 9.4.39. Then p(zZ)h(z) satisfies the differential condition if 89 is sufficiently close to 
1. Therefore, by Theorem 9.4.33, the open set {(z,t) € D x C; |t|? < y(zz)h(z)}, which has a 
C? boundary, is lineally convex; as a consequence also its limit as sq tends to 1, viz. Q itself, is 
lineally convex. 


Proof Using formula (9.47) for b(z), the relation between the inequality (9.60) used in the proof 
of Proposition 9.4.38 and the tangent condition is easy to establish. We observe that 
|b(z)| = |z — h(z)/hz(z)| > |z| if and only if Re2zh,(z)/h(z) <1. Thus if Q satisfies the 
tangent condition, then h satisfies (9.60) for some a < 1 and all z in some sufficiently narrow 
annulus \/s9<|z|<1.4 


Define 
A= sup 2h) anda(sj)= sup Re | ee. 
jaicr! PO) Jo< zl <1 @) 


If A<1 we are done; otherwise we can choose s)9<1 so close to 1_ that 
(1 — s)?<2(1 — a(so))/(A? — 1). Proposition 9.4.38 can be applied and shows that y(zZ)h(z) 
satisfies the differential condition. 


We shall now prove that it can never happen that Re 2zh,(z)/h(z)>1 for any z with |z|<1. 


Proposition 9.4.43 If h € C? (D), h > 0, satisfies the Behnke—Peschl differential condition 
(9.59), then Q satisfies the tangent condition (9.63). 


Proof Let us define 


bo(r) = inf 


l2|<r 


b(z)|, 0<r<il. 


This is a decreasing function and it is continuous where it is finite. The tangent condition for 
2, = {(z,t) € D(0,r) x C; |t|? < h(z)} means precisely that bo(r) > r. It is clear that the 
condition is satisfied for a very small r. Indeed, b(0) = —h(0)/h,(0) is either « or a non-zero 
complex number; in view of the continuity, |b(z)| > r if |z|<r and ris small enough. 
If the tangent condition is satisfied for a particular (2,, then by Proposition 9.4.42 the set (2, 
is lineally convex, so Lemma 9.4.41 can be applied and shows that bo(r)>r + e€, where 
inf R 
“<— >0. 


2sup |R, 


|2|<1 


4Here we could remark that it would be enough to require that b(z) ¢ @ only for all z € Ow, supposing that h € C?(@). The 
stronger condition used in Definition 9.4.40 is however easier to handle in the proof of Proposition 9.4.43. 


We know that bo(r) > r for small values of r, and we have just seen that if bo(r) > 7, then also 
bo(r)>r + €, for a positive ¢ that does not depend on r. Therefore that function cannot assume 
any value in the interval |r,r + |: it must satisfy bo(r) > r all the way up to and including 
r = 1. This means that O satisfies the tangent condition. 


Theorem 9.4.44 Leth € C?(D), h > 0, satisfy the Behnke—Peschl differential condition (9.59). 
Then the open set Q = {(z,t) € D x C; |t|? < h(z)} is lineally convex. 


Proof If h € C?(D) with h > 0 in D we see from Proposition 9.4.43 that Q satisfies the tangent 
condition, so that Proposition 9.4.42 can be applied. In the general case with h € C?(D), h > 0, 
we apply this result to a smaller disk rD, r < 1, to conclude that the domain over rD is lineally 
convex. Then we let r — 1. 


Lr 
9.5 Weak Lineal Convexity 


We start this section with a general presentation of weak lineal convexity. We then discuss local 
variants of this property. 

A locally weakly lineally convex open set with boundary of class C'! is also (globally) weakly 
lineally convex provided that it is bounded. But, as shown by Yurii Zelinskii, this is not true for 
unbounded domains. The purpose here is to construct explicit examples, Hartogs domains, 
showing this. Their boundary can have regularity C1! or C™. 

Obstructions to constructing smoothly bounded domains with certain homogeneity properties 
will be discussed. 


9.5.1 Introduction 


After the main definitions about variants of lineal convexity, we shall approach the comparison 
global vs. local. In my paper (1998) I proved that a differential condition that I called the 
Behnke-—Peschl differential condition implies that a bounded and connected open subset of C” 
with boundary of class C’ is weakly lineally convex. The proof relied on a result by YuZakov and 
Krivokolesko (1971a, 1971b), proved also in (Hérmander 1994: Proposition 4.6.4). 

Yurii Zelinskij (2002a, 2002b) published an example of an unbounded set that is locally 
lineally convex but not lineally convex. His example is not very explicit. We shall construct here 
an explicit example—actually a Hartogs domain, which has the advantage of being easily 
visualized in three real dimensions. We construct domains with boundary of class C1 and a 
certain homogeneity property (Example 9.5.13), and show that this cannot be done with a 
boundary of class C’ (Proposition 9.5.18). However, the boundary can be of class C’™ if the 
homogeneity requirement is dropped (Example 9.5.14). 


9.5.2 Lineal convexity 


The property of being lineally convex was defined in Definition 9.4.1 on page 279. To wit: 


Definition 9.5.1 A subset of C” is said to be lineally convex if its complement is a union of 
complex affine hyperplanes. 


To every set A there exists a smallest lineally convex subset (A) that contains A. Clearly the 
mapping w: A(C") > A(C”), where A(C") denotes the family of all subsets of C” (the 
power set), is increasing and idempotent, in other words an ethmomorphism (morphological 
filter). It is also larger than the identity, so that wis a cleistomorphism (closure operator) in the 
ordered set A(C”). 

This kind of complex convexity was introduced by Heinrich Behnke (1898-1997) and Ernst 
Ferdinand Peschl (1906-1986). I learnt about it from André Martineau (1930-1972) when I was 
in Nice during the academic year October 1967 through September 1968. See Martineau's papers 
(1966, 1967, 1968), also in (Guvres de André Martineau 1977). 

Are there lineally convex sets which are not convex? This is obvious in one complex variable, 
and from there we can easily construct, by taking Cartesian products, lineally convex sets in any 
dimension that are not convex. But these sets do not have smooth boundaries. Hérmander 
(1994:293, Remark 3) constructs open connected sets in C” with boundary of class C’ as 
perturbations of a convex set. These sets are lineally convex and close to a convex set in the C 
topology, and therefore starshaped with respect to some point if the perturbation is small. Also 
the symmetrized bidisk 


{(z1 + 29, 2122) € C?; |z1|, |Z2| < 1}, 


studied by Agler & Young (2004) and Pflug & Zwonek (2012), is not convex—not even 
biholomorphic to a convex domain (Nikolov et al. (2008)—but it is starshaped with respect to the 
origin (Agler & Young 2004: Theorem 2.3). So we may ask: 


Question 9.5.2 Does there exist a lineally convex set in C", n>2, with smooth boundary that is 
not starshaped with respect to any point? 


We shall return to this question in SubSection 9.5.10. 
9.5.3 Weak lineal convexity 


Definition 9.5.3. An open subset Q of C” is said to be weakly lineally convex if there passes, 
through every point on the boundary of Q, a complex affine hyperplane which does not cut Q. 


It is clear that every lineally convex open set is weakly lineally convex. The converse does not 
hold. This is not difficult to see if we allow sets that are not connected: 


Example 9.5.4 Given a number c with 0 < c < 1, define an open set 2, in C? as the union of 
the set 


{z= (z1 + iyi, 22 + iys) € C?;¢< x1 <1, ly | <1, |x2| <o¢, |y2| < ch 


with the two sets obtained by permuting 21, £2 and yy. Thus (2, consists of six boxes. It is easy 
to see that it is weakly lineally convex, but there are many points in its complement such that 
every complex line passing through that point hits 2Q.. 


Any complex line intersects the real hyperplane defined by y; = 0 in the empty set or in a 
real line or in a real two-dimensional plane, and the three-dimensional set {z: y1 = 0} NL, is 
easy to visualize. 


It is less easy to construct a connected set with these properties, but this has been done by 
YuZakov & Krivokolesko (1971b:325, Example 2). See also an example due to Hérmander in the 
book by Andersson, Passare & Sigurdsson (2004:20-21, Example 2.1.7). 

However, the boundary of the constructed set is not of class C!, and this is essential. Indeed, 
YuZakov & Krivokolesko (1971b:323, Theorem 1) proved that a connected bounded open set 
with “smooth” boundary is locally weakly lineally convex in the sense of Definition 9.5.8 below 
if and only if it is lineally convex. It is then even C-convex (1971b:324, Assertion). See also 
Corollary 4.6.9 in (Hérmander 1994), which states that a connected bounded open set with 
boundary of class C! is locally weakly lineally convex if and only if it is C-convex (and every C- 
convex open set is lineally convex). 

There cannot be any cleistomorphism connected with the notion of weak lineal convexity for 
the simple reason that the property is defined only for open sets. We might therefore want to 
define weak lineal convexity for arbitrary sets. We may ask: 


Question 9.5.5 Is there a reasonable definition of weak lineal convexity for all sets which keeps 
the definition for open sets and is such that there is a cleistomorphism associating to any A C C” 
the smallest set that contains A and 1s weakly lineally convex? 


Question 9.5.6 The operation L +> L 22 associating to a complex line L its intersection with 
an open set Q has continuity properties that seem to be highly relevant for weak lineal convexity. 
Here the family of complex lines can arguably have only one topology, but for the family of sets 
LI 2 there is a choice of several topologies, especially if Q is unbounded. 

Can an interesting theory be built starting from this remark? 


9.5.4 Local weak lineal convexity 


Definition 9.5.7 We shall say that an open set QC C” is locally weakly lineally convex if for 
every point p there exists a neighborhood V of p such that 92] V is weakly lineally convex. 


Obviously, a weakly lineally convex open set has this property, but the converse does not hold, 
which is obvious for sets that are not connected: Take the union of two open balls whose closures 
are disjoint. Also for connected sets the converse does not hold as we showed in Example 9.4.8. 
In that example it is essential that the boundary is not smooth. 

Zelinskij (1993:118, Example 13.1) constructs an open set which is locally weakly lineally 
convex but not weakly lineally convex. The set is not equal to the interior of its closure. 


Definition 9.5.8 Let us say that an open set Q is locally weakly lineally convex in the sense of 
YuZakov and Krivokolesko (1971b:323) if for every boundary point p there exists a complex 
hyperplane Y passing through p and a neighborhood V of p such that Y does not meet VQ 92. 


Zelinskij (1993:118, Definition 13.1) uses this definition and calls the property 
voKranonas aunetinas eunyrxaocmd (lokal'naja linejnaja vypuklost’). (Ulokal naja linejnaja vypuklost). 


As we shall see, this property is strictly weaker than the local weak lineal convexity defined 
above in Definition 9.5.7. 

Hormander (1994:Proposition 4.6.4) and Andersson, Passare & Sigurdsson (2004: 
Proposition 2.5.8) use this property only for open sets with boundary of class C'. Then the 
hyperplane Y is unique. 

For all open sets, local weak lineal convexity obviously implies local weak lineal convexity in 
the sense of Yuzakov and Krivokolesko. In the other direction, Hérmander's Proposition 4.6.4 
shows that for bounded open sets with boundary of class C', local weak lineal convexity in the 
sense of Yuzakov and Krivokolesko implies local weak lineal convexity (even weak lineal 
convexity if the set is connected). 

Nikolov (2012:Proposition 3.7.1) and Nikolov et al. (2013:Proposition 3.3) have a local result 
in the same direction: If 0 has a boundary of class CK 2<k<oo, and MN Be (p, 7), where p is a 
given point, is locally weakly lineally convex in the sense of Yuzakov and Krivokolesko at all 
points near p, then there exists a C-convex open set w (hence lineally convex) with boundary of 
class C* such that wM Be(p,r’) = Q2N Be(p,r’) for some positive r. 

However, in general, the two properties are not equivalent: 


Example 9.5.9 The bounded connected open subset Q" of C*, taking r = 2, which was defined 
in Example 9.4.8 on page 281, has Lipschitz boundary and is locally weakly lineally convex in 
the sense of Yuzakov and Krivokolesko but not locally weakly lineally convex. While {2" is 
locally weakly lineally convex for 2 <r < / 5, the set Q? is not locally weakly lineally convex: 
The point (0,2) does not have a neighborhood with the desired property. But it does satisfy the 
property of YuZakov and Krivokolesko. 


9.5.5 Approximation by smooth sets 


Let A; C C™ be two lineally convex sets in C”, 7 = 1,2. Then it is easy to see that their 
Cartesian product A, x Ay C C™*™ is lineally convex. In particular, if ny = nz =1, then 
every Cartesian product in C* is lineally convex. However, these sets cannot always be 
approximated by lineally convex sets with smooth boundaries. 

If 2;, 7 = 1,2, are convex open sets, then 2 = 92 x (22 is convex and can be approximated 
from within by convex open set (2/.} with C' boundaries, Qj.) 7 Qas € \, 0. 

But if we let (2; be an annulus and {22 a disk, e.g., 


QN=M x 2, = {z2EC?;1< |z| < 3, |z2| < 1}, 


then it cannot be approximated by smooth weakly convex sets from the inside as we shall see in 
the next proposition and its corollary. 


Proposition 9.5.10 Let @ be a nonempty bounded open subset of R? with boundary of class C. 


Suppose that inf x£1| > 0. Define a Reinhardt domain Q as 
Lew 


Q={zEC?’; (|z1|, |z2]) ew}. 


Then Q is not locally weakly lineally convex. 


Proof. Take a point g = (q1,q¢2) € @ with q, < 0, q2>0. Denote by 27 the set of all re 2 
such that x; > 0 and z2 > 0, and by L, the complex line of equation z; — q2 = a(z1 — q1), 
a>0. For a = 0, the line cuts 2* in (—qu, q2); for large a it does not cut 2+. Now choose the 
smallest a such that L,, does not cut 2+. Then L, contains at least one point p € 02, and L, is 
the tangent plane of 02 at p. Since this line meets © in gq, O is not weakly lineally convex. But 
we can say more: It is not even locally weakly lineally convex. To see this, first note that 
a = (p2 — q2)/(pi — q1) > 0. Then there are points z € 2 belonging to the tangent at p 
arbitrarily close to p. Indeed, since a is positive, a point z satisfying 


|21| > pi and |z2| < po 


belongs to if it is close enough to p. In terms of z, this means that 


|z1| > pi and |q2 + a(z1 — q1)| < pe; 


in other words that z1; ¢ D<(0,p1) and that z: € De(c1,1r1), the open disk with center at 
C1 = qi —q2/a and radius rr; = po/a=p,—c;. Since r1) = pi— ci, there are points 
z. € De(c1,171)\D<(0, pi) which are arbitrarily close to p;. 


Corollary 9.5.11 A Reinhardt domain 
Q= {z€ C*; 17, < |z1| < Ri, |z2| < Ry} 


with r; > 0 is lineally convex but cannot be approximated by lineally convex domains with 
boundary of class C?. 
Proof. If a domain §2;.; approximates © from the inside in the sense that 


Qe} CMC Me] + Bz(0,¢), 


then there is also a Reinhardt domain with this property: We may construct such a set by 
averaging over all rotations. 
We can now apply the proposition to 92/,). 


9.5.6 The Behnke—Peschl and Levi conditions 


We refer to SubSection 9.4.5 for the definitions of the real and complex Hessian, the Levi form 
as well as the Levi condition and the strong Levi condition. Moreover, we defined there the 
Behnke—Peschl differential condition and the strict Behnke—Peschl differential condition. 

The Behnke—Peschl differential condition says that the restriction of the real Hessian to the 
complex tangent space at any boundary point shall be positive semidefinite; for the strong case, 
positive definite. 

Because of the different homogeneity of H© and L, the inequality Re H . + L,20 is 
equivalent to L>|H © |: The inequality L>|H © |20 shows that the Behnke—Peschl differential 
condition implies the Levi condition. 

In my paper (1998) I proved that a bounded connected open set with boundary of class C’ is 
weakly lineally convex if it satisfies the Behnke—Peschl differential condition. 


That this condition is necessary for weak lineal convexity was known since Behnke and 
Peschl (1935); the sufficiency was unknown at the time. 
9.5.7 Yuzakov and Krivokolesko: Passage from local to global 
Let us quote the part of Proposition 4.6.4 in (Hérmander 1994) which is important for us: 
Proposition 9.5.12 Let QC C” be a bounded connected open set with boundary of class C+ and 


assume that Q is locally weakly lineally convex in the sense of Yuzakov and Krivokolesko. Then 
Q is weakly lineally convex. 


The result was proved by YuZakov & Krivokolesko (1971a, 1971b) under the condition that the 
boundary is “smooth.” 

9.5.8 A new example 

We shall construct explicit Hartogs domains here with the properties mentioned in Zelinskij's 
example. We start with the simplest. 


Example 9.5.13 Define a function py? : C + R by 


2? —y?, 21<0or <0; 
y°(z1) = —x} +y?, 0<yi<21; 
a4 — yf, O<r1<y1. 


Then Qyo = {z € C?; 1+ y°(z1) + |z2|2 <0} has boundary of class C'' and is locally 
weakly lineally convex but not weakly lineally convex. 


3 +i 
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FIGURE 9.2 
The set Q,° 9 {z €C?: zx = oO}. 


The properties of the set in this example will be discussed now and the properties will be seen to 
hold from Proposition 9.5.15. 


The tangent plane at a boundary point p= (p1,p2) with Rep; >0, Imp; > 0, and 
(Re pi)” > (Im p1)? +1, has the equation —pi(z1 — pi) + po(z2 — p2) =0 and it passes 
through the point gq = (pi —|p2|?/p1,0). Choosing p=(3+i,V7) we — get 
q= (+ + itt, 0) € Nye, proving that 2, is not lineally convex. 

We note that the tangent plane at a boundary point p with Re p;<0 or Im p;<0 is contained 
in the complement of 2,°; in particular, it hits the plane z2=O at the point 
g = (p1/|pi|?, 0) ¢ Aye. We also note that the part of 2,,> where 0 < x1 < yj is convex, so any 
tangent plane of this part does not intersect it. Similarly, the part where 0 < y; < 2%, is convex. 
Therefore {2¢ is the union of two lineally convex sets, taking the subsets where 71 < max(y1, 0) 
, and y; < max(21, 0), respectively. 

When 21 <0 or y; <O we get 9$ (21) =—%, ¥%,,(21) =9, yz, (1) = —1; when 
0 < yi<a1 we have vy? (21) = —21, 92,2,(21) = —1 and p> 7, (21) = 0; when 0 < 21<y1 we 
have 9§, (21) = 21, 9%,,,(21) = 1 and y$ , (21) = 0. In all three cases |(p,,2,| — Paz, = 1. An 
application of Proposition 9.5.15 below now gives the result, except that it does not give anything 
at the exceptional points, where the function is not of class C'™, i.e., those with y; = 0, 1; > 0 
or x; = 0, y1 > 0. However, we have already seen that at these points, the tangent plane does not 
cut Qye. 

The boundary of §2, is not of class C at the points where y; = 0, x; > Oor z; = 0, y; > O 
. The passage from —x? — y? for y;<0 to —a} + y? for O<y; <2, cannot be made analytically. 

The function g® is not of class C1! at the points where x; = y;, 21 > 0, but this is of no 
consequence, since these points do not belong to the closure of the set it defines. 


We note that the function g° in the example is homogeneous of degree two: 
ip’ (zi) = y°(|z1 |e”) = |z1|?0(t), z1€C,teER. 


It is therefore natural to ask if there is a C'° homogeneous function g with the same properties. 
More precisely, we may ask for functions y : C — R which yield a locally weakly lineally 
convex domain that is not weakly lineally convex in four different cases. 


1.1. Is thereaC®™ function y with these properties? 

1.2. Is there a homogeneous C'® function g with these properties? 
2.1. Is there an analytic function g with these properties? 

2.2. Is there a homogeneous analytic function g with these properties ? 


As we shall see, the answer to the first question is in the affirmative (Example 9.5.14). But the 
answer to Question 1.2 is in the negative (Proposition 9.5.18). 


Example 9.5.14 Now define y* : C + R by 


—x? + x(y1), 21>y1; 
—y, + x(x), L1<Y1; 


where x € C'™(R) is a function of one real variable such that x' is convex and which satisfies 


—yitp, WWS- F3 
x(y1) = 2 1 


The convexity of x' implies that 2|y1|<|x'(y1)|< max(2|yi|,1) with equality to the left for 


lwi|>s- This implies that we must have + < x(+) — x(—¥) < 1, and we can actually choose 
x so that x(5) — x(—+) is any given number in that interval. 


For definiteness we now choose p = —F, o = 0, x( , )—x( , = 4 , which implies that 


yp? — t< y*<yp®, that 2. contains 2,0, and that the set of points z € Qy« with Re u2>y and 
Im 2,2 + is unchanged compared to 92°. We choose x as a suitable third primitive of 


x!" (y1) = Cexp(1/(y: — c) — 1/(y1 +0), —c<y<c, 


for a number c, 0 < c<, and a positive constant C, taking x'"(y1) equal to zero when |y;|>c. 
This implies that x' is even and that x(0) = , pt , C= . . Then 


Qo = {z € C?; 1+ y*(z1) + |z2|? < Oo} 


has boundary of class C'® and is locally weakly lineally convex but not lineally convex, since, 
just as for Q,e, the tangent plane at the boundary point p = (3 +i, V7) passes through 
q= (2 +it,0) € MQ. 


The properties mentioned in these two examples will follow from the next proposition and its 
corollary. 


Proposition 9.5.15 Let y: C — R be a function of class C*, k = 2,3,...,00,w (C° denoting 
the family of all real analytic functions), and define an open set in C? as 


Q, = {2 € C?; 14 v(z1) + |z2|? < O}. 
We assume that 
~z, # Owherever y = —1, 
(9.65) 
and that 
|? 


(—~ — 1)(|%z,2,1 — Pxz,)<lz,|* inthesetwhere — p — 130. 


(9.66) 
Then 92, has boundary of class C* and satisfies the Behnke—Peschl differential condition at every 


boundary point. If the inequality is strict at a certain point, we get the strict Behnke—Peschl 
differential condition at that point. 


Proof In Lemma 9.4.26 on page 288 I described the domain by an inequality of the form 
|z2|? < h(z1) and found that the Behnke—Peschl differential condition takes the form 
h(hz.z, + |Reaz|)<|Rz,|?, which, with h(z1) = —y(z1) — 1, yields (9.66). 


Corollary 9.5.16 Let g have the form y(z1)=—2?+x(yi) for «21>y, and 
y(z1) = —y7 + x(a1) for y1>21. We assume that x € C*(R), k>2, with —2< x" and such that 
x(y1) > —1 when y'(y1) =0. Then the conclusion of Proposition 9.5.15 holds under the 
assumption 


tx'(y1)? +x(y1) +120, meR. 
(9.67) 


Proof The condition (9.65) is satisfied, since the gradient of g in this case vanishes only when 


z,=0 and x’/(y1) =0. Then 1+ y(z1) + |ze/? =1+yx(y1) + |z2/? > 0, so z= (iy, 22) 
cannot be a boundary point of 0. 
Condition (9.66) reduces to 


(a? — x(y1) — 1)(|-$ — $x"(1)| + $ — $x"(y1))<| — 21 — Fix'(yi)|? = 22 + $x'(y1)", 


provided x? — x(yi) — 150. If -2<y", we have 


4 — dx"(u)| +4 - dey) = $+ dx") + $40") = 1, 


which gives (9.67). We then see that in this case the inequality holds also if ae —x(yi) -1<0. 


In Example 9.5.14, the defining function 1 — a? +x(yi) + |z2|? has nonvanishing gradient 
everywhere since 7’ > 0 everywhere. Smoothness follows. 

The function g* is not of class C'® in the set where x; = y1, x; > 0, but again this is 
unimportant since these points do not belong to the closure of (2,+. An application of Corollary 
9.5.16 now gives the result. In fact, with the choice of p = —4, ao = 0, we need only note that 
x(yi)> — y; = i everywhere, and that x’(y1) >2|y1|, so that 


4x! (yi)” + x(y1) +122 > 0, L121, 
thus with strict inequality in (9.67) and (9.66); similarly for x;<y}. 


Remark 9.5.17 It might be of interest to understand where the proof of Hérmander's Proposition 
4.6.4 quoted above breaks down in the unbounded case. An important step in the proof is to see 
that, if we have a continuous family (Le) se10,1 of complex lines, the set T of parameter values t 
such that L,M 92 is connected is both open and closed. Thus, if0 € T, then alsol € T. We shall 
see that closedness is no longer true for the sets in Examples 9.5.13 and 9.5.14. 

Define complex lines 


Ly = {2€ C*; 22 =t(z1-1-i)}, te [0,1], 


which all pass through (1+i,0) ¢ Q,*. Then Ly,» is connected for 0<t <1 while 
[1M Qy« is not. 
We shall first see that Ly ™ Q,» is disconnected. If z € Ly 1 Qy« and x, <0 or yi <0, then 


f(z) = 1+ * (21) + |z0|?>1 — Jaa? + p + Jz1 — 1 —i]? = 3 + p— 2(21 +: 1). 


Since we have chosen p = —+, the quantity f(z) can be negative only if x1 + y; > 0, which 
implies that z, satisfies either x; > |yi| or y1 > |x|. Therefore the real hyperplane of equation 
£1 = y1 divides L1 M 2, into two sets, which are nonempty since (2,1 — i) and (2i,i— 1) both 
belong to Ly, the first with y, < x1, the second with y, > x1, and that both belong to Q,« in 
view of the fact that x(0)<0. 

Next we shall see that Ly 1 Q,* is connected when 0<t < 1. Given t such that 0<t < 1, we 
obtain for z © Ly 1 Qy« with £1, yi<0, 


1+ y*(z1) + |z2|?<1 — (1 — ¢?) [21]? — 2t?(a1 + yr) + 20?. 
This yields the estimate 
1+ y*(z1) + |z2|?<3 + Al z1| —(1- ap leaps 


which is negative when |z;| = R, for a large enough number R,, which depends on t. Obviously 
R, tends to +00 as t — 1, which explains that L, \ Q,« is disconnected. Let I, be the arc in L, 
with |z1| = R, and x; <0 or y;<0, thus contained in Q,.. 

An arbitrary point a € Ly Q,« can be joined to a point in I, by a straight-line segment 
contained in 9,+ and therefore also contained in L, 1 92Q,*. If Rea,<0 or Ima;<0 this follows 
from the fact that the set of points in (2, with argument of z, equal to that of a, is convex; 
otherwise from the fact that the points in 2Q,* with 0<Im z; < Re z, is convex, as is the set of 
points with 0<Re z; < Im2j. 


9.5.9 An impossibility result 


Proposition 9.5.18 Let 2Q,= {z €C*;1+ 9(z1) + |z2|? < Oo}, where gy 1s positively 
homogeneous of degree two and of class C? where it is negative. Then either y is constant and 
QQ, is lineally convex; or y is not constant and (2, is not connected. 


The set 2, in Example 9.5.13 has the properties mentioned here except that its boundary is not 
of class C*: We have a striking contrast between the regularity classes C1! and C. 

Proof. For functions y : C + R which are positively homogeneous of degree two, i.e., of the 
form (21) = |z1|?(t), z1 = |z1 lew € C, t € R, condition (9.66) on 9 takes the form 


2 2 
(r= 1)]-o + 09? + G0y - 40") rte? + 0), 
to hold in the set where —r? — 150; equivalently 


sb-+ (rb — 1) | V+ av? — "] <P, 


From this we obtain, if we divide by rand let rtend to +00, 


(—v) | Vor? + 4b? — w"] <p”. 
(9.68) 


But this condition is also sufficient, which follows on multiplication by 7 and adding the trivial 


inequality 4%) — |e + Ay! — "| <0. 


To get rid of the square root in (9.68) we rewrite it as 


py? (pb? ae 2(—)" _ Arp?) >0, 


where the left-hand side is of degree four. 

We now introduce a function g by defining g(t) as the positive square root of —w(t) if Y(t) is 
negative and as 0 at all other points. Thus g is of class C’ where it is positive, and ~ = —g? 
there. The points where g=0, equivalently w~>0, are not of interest, since for them 
1 + |z1|?¢b(t) + |z2|?>1 > 0, implying that z does not belong to the closure of Q,. 

We get an inequality of degree eight but which is easy to analyze: 


99" (9+ 9")<0. 
(9.69) 


Thus, for each ¢ such that g(t) > 0, either g'(t) = 0 or g(t) + g"(t)<0. If g is zero everywhere, 
i.e., if gis constant, it is known that (2, is lineally convex, in particular weakly lineally convex. 


Wherever g is positive and g is nonzero we get g + g’<0. This implies that any local maximum 
of g is isolated and that there can only be one point where the maximum is attained. Hence, 


unless g vanishes everywhere, g + g’<0 everywhere. We define h = g + g”<O and obtain for 
anyacR 


g(t) = g(a) cos(t — a) + g(a) / sin(t — s)h(s)ds, teR. 


The function g attains its maximum at some point which we may call a, and the formula then 
shows that g(t)<g(a) cos(t — a) for all t with a<t<a + 7/2. In particular, g must have a zero 
in the interval Ja,a + 1/2]. By symmetry, g has a zero t, also in the interval [a — 1/2, a[, hence 
at least two zeros in a period. This means that is not connected, since the union of the rays 
arg z1 = to and arg z; = t divides the z,-plane. 


9.5.10 A set which is not starshaped 


A subset A of a vector space is said to be starshaped with respect to a pointa € A if the segment 
[a,b] is contained in A as soon as b belongs to A. 
In answer to Question 9.5.2 we mention a modification of the set 2,° which is not starhaped. 


Example 9.5.19 Define y' : C > R by 


t —ai — yi, £1 + yi <0; 
~ (21) = 1 2 
—3z(t@1-yi1)°, 21 +4120. 
Then QO, = {z € C?; 1+ p#(z1) + |z2|? <0} has boundary of class C' and is lineally 


convex, but it is not starshaped with respect to any point. 


This set can conceivably be modified to have a boundary of class C'™ like in Example 9.5.14. 
However, it is unbounded. 


Question 9.5.20 Does there exist a bounded set with boundary of class C? which is lineally 
convex but not starshaped? 


FIGURE 9.3 
The set 2549 {z € C?; 2 = oO}. 


Defining a function g : R > R of period 27 by g(t) = | cos 2t| when 0<t¢<z and g(t) = 1 when z < t < 27, we geta 
function which satisfies inequality (9.69) in R\7Z, but not the conclusions we have drawn from it. This function is not of class 


fou (but of class C11), 
Question 9.5.21 What about a Hartogs domain with these properties? 


The set © defined in Corollary 9.5.11 is bounded, lineally convex and not starshaped, but it has 
only a Lipschitz boundary. It cannot be approximated by a lineally convex set with smooth 
boundary. 


9.6 A Differential Inequality Characterizing Weak Lineal Convexity 


Abstract of this section 

Behnke and Peschl introduced in 1935 the notion of Planarkonvexitaét, now called weak lineal 
convexity. They showed that, for domains with smooth boundary, it implies that a differential 
inequality is satisfied at every boundary point. We shall prove the converse here. 


9.6.1 Introduction 


In an article published in the Mathematische Annalen in 1935, Heinrich Behnke (1898-1979) 
and Ernst Peschl (1906-1986) introduced a notion of convexity called Planarkonvexitat, 
nowadays known as weak lineal convexity. They showed that for domains in the space of two 
complex variables with boundary of class C’, this property implies that a differential inequality is 
satisfied at every boundary point. Here we shall prove that, conversely, the differential inequality 
is sufficient for weak lineal convexity. 

Behnke and Peschl (1935:170) proved that for sets with smooth boundary, weak lineal 
convexity is a local property (see Theorem 9.6.2 below). 

Both usual convexity and pseudoconvexity can be characterized infinitesimally. The simplest 
example of such a result is that a C’ function of one real variable is convex if and only if its 
second derivative is nonnegative. More generally, a domain in R” with boundary of class C is 
convex if and only if the Hessian of a defining function is positive semidefinite in the tangent 
space at every boundary point. Similarly, an open set in C” with boundary of class C’ is 
pseudoconvex if and only if the Levi form of a defining function is positive semidefinite in the 
complex tangent space at every boundary point (the Levi condition). 

In analogy with these two classical results, we shall prove in the present section that a 
bounded connected open subset of C” with boundary of class C* is weakly lineally convex if and 
only if the real Hessian of a defining function is positive semidefinite in the complex tangent 
space at every boundary point (the Behnke—Peschl condition). 

It is easy to see that semidefiniteness is necessary. It is also known—indeed, this is the 
Hauptsatz of Behnke and Peschl (1935)—that the corresponding strong condition, i.e., that the 
real Hessian be positive definite, is sufficient. Thus what we have proved is that semidefiniteness 
is sufficient. 

In the case of convexity and pseudoconvexity, the best way to deal with semidefiniteness is to 
approximate the domain by domains which satisfy the corresponding stronger condition of 
definiteness. This is not how we approach the problem here, at least not directly. The idea of 
proof of the main result here is instead to construct Hartogs domains which share a tangent plane 
with the given domain. 


Question 9.6.1 Can a weakly lineally convex domain with smooth boundary be approximated 
from the inside by domains satisfying the strong Behnke—Peschl condition? (For Hartogs 
domains this is known; see Theorem 9.4.33. 


9.6.2 The main result 


To be able to characterize sets by infinitesimal conditions, we shall describe boundaries and their 
curvature using defining functions and the Hesse and Levi forms. We refer to the definitions 
already given in SubSection 9.4.5. 

As noted in the introduction, lineal convexity is not a local condition. Simple examples of 
sets which are locally lineally convex but not weakly lineally convex can be found in Section 9.4. 


However, weak lineal convexity is a local condition for sets with smooth boundary. The precise 
result is as follows. 


Theorem 9.6.2 Let Q be a bounded connected open set in C” with boundary of class C'. 
Assume that for every boundary point a, the closure of the intersection of Q with the complex 
tangent plane at a does not contain a. Then Q is weakly lineally convex. 


For sets in C? or P? with boundary of class C’, this was proved by Behnke and Peschl 
(1935:170). For a proof under the hypotheses stated here, see (H6rmander 1994: Proposition 
5.6.4.) See also (Andersson, Passare & Sigurdsson 2004: Proposition 2.5.8). We shall need this 
result in our proof. 

We recall two lemmas from Section 9.4: Lemmas 9.4.24 and 9.4.25, both due to Behnke and 
Peschl (1935: Theorems 7 and 8); local weak lineal convexity is called Planarkonvexitat im 
kleinen by them. Cf. also (Zinov ev 1971), (Hérmander 1994: Corollary 5.6.5). 

Combining Lemma 9.4.25 and Theorem 9.6.2 we can deduce that the strict Behnke—Peschl 
differential condition (9.53) at all boundary points is sufficient for weak lineal convexity. This is 
the Hauptsatz of Behnke and Peschl (1935:170) (for sets in C? or P?). We now state our main 
result, that in fact also the weaker condition (9.52) is sufficient: 


Theorem 9.6.3 Let Q be a bounded connected open set in C” with boundary of class C?. Then Q 
is weakly lineally convex if and only if Q satisfies the Behnke—Peschl differential condition 
condition (9.52) at every boundary point. 


If Q is locally weakly lineally convex, has a C! boundary, and in addition is bounded, then 0 is 
also C-convex and lineally convex. This follows from (Andersson, Passare & Sigurdsson 2004: 
Proposition 2.5.8), who consider sets in projective space. I do not know how their result can be 
applied to unbounded domains in C” with smooth boundary; such domains are not necessarily 
smoothly bounded in P”. 


9.6.3 Results for Hartogs domains 


Lineal convexity for Hartogs sets is easier to handle than in the general case. For some results, 
see Section 9.4, in particular Theorems 9.4.33 and 9.4.44. 


Proposition 9.6.4. Let Q be an open set in C” and define 
Qy = {z € ©"; (21,...,Zn-1, AZn) € 2 for all A € C with |A|<1}. 
(9.70) 


This is the largest complete Hartogs set contained in Q. If Q is lineally convex, then Qy is 
lineally convex; similarly for weak lineal convexity. If 02 is of class C? except perhaps where 
Zn = 0, then so is the boundary of Qy at all points z with z, # 0 and satisfying the condition 


(9.71) 


where M is a bound for the second derivatives pz,z, and Pz,z,. If, in addition, Q satisfies the 
Behnke-Peschl differential condition (9.52) at all boundary points with z, # 0, then so does Qy 
at all boundary points with z,, # 0 satisfying (9.71). 
Proof If © is lineally convex, then also (2y, as an intersection of lineally convex sets, has this 
property: 

QH= My, where 2) = {2 € CO”; (21,.--,;2n—1, AZn) € ah, 


|A|<1 


Assume now that © is only weakly lineally convex, and let a point a on the boundary of 24 
be given. Then for some 2 with |A| = 1, a is on the boundary of (2) defined above, and a 
hyperplane through a which does not intersect (2, does not intersect (27 either. (The argument is 
valid for all a; if a, = 0 we even have a € 02) for all A.) 

If p defines 0, then 


pu(z) = supp(21,...,Zn—1,€82n) 
6ER 


(9.72) 
defines 27 in a neighborhood of its closure. Define 
Mer) (z,0)€ C"x R. 


DO Fisss ny Pee) SO Aiseers eae! 


We can calculate 


yo =—2Im (pz,e% Zn); 
yee = —2Re (92, 2n) — Re (Penz0 Zn) + Ween 


rae 
The value of @ which defines the supremum in (9.72) solves the equation y; = 0, and the 
implicit function theorem can be applied if p%, 4 0 there. This condition is fulfilled if 


|Re (02,0? 2n)| > 2M|zn|", 
(9.73) 


where M is a bound for the second derivatives of p as defined in the statement of the proposition. 
However, when Lo = 0, the expression Pz, Zn is real, so that (9.73) simplifies to (9.71). The 
implicit function theorem then says that the boundary of (27 is as smooth as that of where the 
condition is satisfied. 

Now assume that 0) satisfies the Behnke—Peschl condition at a boundary point a of Qy with 
ay, # 0. Then a is on the boundary of some 2), |A| = 1, as already noted above. Consider the 
functions 


yr(s) = pr(a+ st), pu(s) = pu(a-+ st), sER, t€Te(a), 


where p)(z) = p(21,.--,;2n—1,AZn), the defining function for 2) obtained by rotating p in the 
last coordinate. 

The Behnke—Peschl condition holds for 2), which means that (~ ,)"(0)>0. Now yH>9 
and both functions vanish at the origin, which implies (y#)"(0)>(y,)"(0). Thus the condition 
holds for (277. This completes the proof. 


In an application of this proposition in the next subsection we shall let 0 be defined near an 
arbitrarily given point by an inequality y, < f(z’,x,) for some real-valued function f of n — 1 
complex variables and one real variable. Then p(z) = yn — f(z’, 21) is a defining function for 0 
near the given point. (Here x, = Re2Zn, Yn =Im2Zn, and 2’ = (21,...,2n-1).) We see that 
p= —F(fepn + i), so that |Pzn >F. Moreover 


P2ntn = PZntn = —fentn: 
This implies that a sufficient condition for (9.71) to hold is 
Clz,| <1, 
(9.74) 
where Cis a bound for f, ,.,. 


Remark 9.6.5 Condition (9.71) has a simple geometric meaning. With the defining function 
pP(z) = yn — f(z’, an) it says that the intersection of the boundary of Q with the subspace 
z' = constant has smaller curvature than the intersection of the boundary of Qy with the same 
subspace where the two boundaries meet. For simplicity we shall use the stronger condition 
(9.74) instead. 


9.6.4 Proof of the main result 


We shall now prove Theorem 9.6.3. In view of Theorem 9.6.2 it is enough to prove that the 
complex tangent plane a + Tc(a) does not cut 0 near a. We shall assume that a + Tc(a) cuts 0 
in a point b and then show that this leads to a contradiction if bis close to a. 

First of all we may assume that n = 2 by looking at the two-dimensional affine complex 
subspace that contains a, b and a third point on the normal to 02 through a. We may also assume 
that the coordinate system is chosen so that a = 0 and the real tangent plane a + Tr(a) has the 
equation Im z2 = 0. We recall that both weak lineal convexity and the Behnke—Peschl condition 
(9.52) are invariant under complex affine mappings. The complex tangent plane at a then has the 
equation z2 = 0, so that bg = 0. We shall consider a neighborhood W of a such that three 
conditions are satisfied. Let 


W= {z EC’; |21| < Ri, |z2| < R2} 
and let V be its intersection with C x R: 


V= {(21, £2) ECx R; \z1| < Ry, |x| < Rp}. 


The three conditions are: 


(A) First of all the set Q shall be defined in Wby an inequality Im zz < f(z1, Re z2) 
for some function f which is of class C’ in a neighborhood of the closure of V. 


(B) Next we shall assume that condition (9.74)) is satisfied for all z € W with some 
margin: 


enero, 


R2sup 5 
V 


fry, 


(This is to allow a change of coordinates later.) 


(C) Third, R; shall be so small that MR, + C(1+ M?)R? < 4Re, where 
M= SCR, and Cis defined below. 


To satisfy these conditions we have to specify the numbers R), R, and C. We first choose R, and 
R, so that (A) and (B) hold, and then define a constant C as follows. Since fis a function of class 


C defined in a neighborhood of the closure of V and with vanishing derivatives of order up to 
one at the origin, there exists a constant C such that 


| f(z1, x2)| <C(|z1|? + 23), 
\fe(Z1,%2)|  <C(|z1| + |x|), and 
| Ferges (215 @2)| < 


Q 


for all (21, 22) € V. We finally shrink R, if necessary to make (C) hold. 

With the choice of coordinate system we have made, the normal at a is the y-axis. Let c be a 
point on that axis with Imcez < 0; it is convenient to take c = —+FiRo. Thus c = (0,c2) and 
Ic| = —Imc2 = +R. The circle in the plane z; = 0 with center at c and radius |c| passes 
through a and is tangent to the x>-axis at that point. 


We shall prove that f(b)<0 (hence that b ¢ §2) for all b with |b1| < Ri. Assume the contrary: 
f(b) > 0. Consider the plane z; = b; and the graph of frestricted to that plane. Draw the normal 
to the graph of f(b1, - ) through the point z2 = if(b1, 0) in the z,-plane. This normal intersects 


the line y2 = Imcy at a point, which we call p,. Define p, = b; and p = (pi, p2) € C?. The 


slope of the normal is determined by the slope of the graph at z; = b1, z2 = 0, ie., by fx, (b1, 0). 
This derivative can however be controlled: we know that f,,(b1,0) is not more than C|b,| in 
modulus. The distance between p and c is 


P— Cl = |Ja.(1, c| + f(91,9))<C |61|( 7ite + 1|°)S yO ft2]61 |, 
|p — ¢| = | fr,(b1, 0) |(e] + f(b1, 0))<C|bi](4+.R2 + Clb |?)<4CR2|b1| 


where the last estimate is a consequence of (C). Thus |p — c|<M|b;| with M = +CRy. 
We have constructed a disk Dp in the plane z; = 0 with center at c, and with z2 = 0 on its 


boundary, and now let D, be the disk in the plane z, = b; with center at p, and if(b,,0) on its 
boundary (and therefore containing z2 = 0): 


Do = {z eC: 2; = 6, |Z2 — C| < le] }; 
Dy = {z€ C’; 2 = bi, |z2 — pal < |if(b1, 0) — pal}. 


Both disks are moreover contained in (21 W. For Dp this is obvious from the construction; for 
D, this can be seen as follows. The center of D, is p, and its radius r, is |if(b1) — p2|. The disk 
is contained in Wif |po| + r1<Rp. This inequality follows from the estimates we already have: 


|p2| + 71<2|p2| + C]bi|?<2|eo| + 2|p2 — e2| + Clbi|?< Ro + 2MR, + CRi< Re, 


where the last inequality follows from (C). Thus D; C W. That D,; C 2 now follows from (B); 
cf. Remark 9.6.5. 

If we construct a Hartogs domain by rotating Q around an axis which passes through c and p, 
then this Hartogs domain will have a on its boundary and contain b. This is precisely what we 
shall do. 

We introduce new coordinates (w1, w2) so that the w,-axis, i.e., the plane w2 = 0, passes 
through c and p. The w,-axis need not be changed. This means that the new coordinates shall be 
defined as 


wi = 21, We = 22 — Co — (po — C2) 21/01. 


Indeed z=c gives w=0 and z=p yields w = b = (01,0). We now define Qy in the w- 
coordinates. The tangent plane with equation z,=0O has_ the equation 
W2 = —C2 — (p2 — C2) w/b; and is also the tangent plane to 027 at the point w = (0, —cz). It 
intersects 27 at the point z = b, i.e., w = (b1, —p2). That this point is an element of Qy follows 
from the construction of Dy. 


We shall now apply Theorem 9.4.33 to Qy over the disk |w | < R, in the w,-plane. To be 
able to do so we have to check that there is a point of 7 over every point w, with |w;| < Rj, or 
equivalently that (wi, 0) € Qz for all w with |w:| < Ri. 

In the new coordinate system, the inequality defining becomes 


Im w2 < —Imcp — Im (pe — c2)w1/b1 + f(wi, Re we + Re (p2 — c2)wi/b1). 
Denote the right-hand side by g(w1, Re w2). In particular 
g(wi, 0) = —Imcz — Im (pp — c2)wi/b1 + f(wi, Re (p2 — c2)w1/b1). 
Recalling the estimate |p2 — c2|<M|bj| above, we get 
g(w1,0)>4R2 — M|w;| —C(1+ M?)|w;|?>+Ro — MR, —C(1+ M?)R? > 0, 


the last inequality coming from (C). This ensures that every point (w , 0) with |w1| < Rj lies in 
© and therefore also in (2;7. 

We know that 27 satisfies the Behnke—Peschl differential condition at all boundary points if 
the condition in the w-coordinates corresponding to (9.74) is valid. Note that 
|we — 22 + c2|<M|z| independently of the choice of b € W, from which we deduce 


lws|<|z2| + ~Re+ MRi<$ Ro. 


The second derivative of g with respect to Re wg is the same as the second derivative of f with 
respect to £2 = Re Z2, so from (B) we can conclude that the condition (4.5) is satisfied also in 
the w-coordinates for all points w € Oy with |wi| < Ry. 

It now follows from Theorem 9.4.44 that Qy is lineally convex, which contradicts the fact 
that the tangent plane at the point w = —c intersects Qy in w = (b1, —p2). This completes the 
proof. 


PR 
9.7 Generalized Convexity 


Abstract of this section 
Inspired by mathematical morphology we study generalized convexity and prove that certain 
subsets of Hartogs domains are convex in a generalized sense. 


9.7.1 Introduction to this section 


By the Hahn-Banach theorem, an open convex set in R” is an intersection of open half-spaces; 
its complement a union of closed half-spaces. What if we replace the latter by balls? We shall 
study here a kind of generalized convexity where a set is called concave if it is a union of closed 
balls; its complement thus being an intersection of complements of closed balls. This will be 
done in particular for Hartogs domains which are lineally convex. 

Lineal convexity is a kind of complex convexity intermediate between usual convexity and 
pseudoconvexity. More precisely, if A is a convex set in C” which is either open or closed, then A 
is lineally convex (this is true also in the real category), and if 0 is a lineally convex open set in 
C”, then 0 is pseudoconvex. 

As mentioned on page 263, there are several different notions of convexity related to lineal 
convexity. 

The main results are presented in Subsections 9.7.8 and 9.7.10. It is shown there that certain 
subsets of Hartogs domains have convexity properties originating in mathematical morphology. 
We also study external tangent planes of sets that do not necessarily have a smooth boundary. 


9.7.2 Hyperplanes, tangent planes and multifunctions 


Hyperplanes are affine subspaces with real or complex codimension 1, and they will play an 
important role in the sequel. 

To any real hyperplane Yin C” and every point a € Y there is a unique complex hyperplane 
Yjq| that contains a and is contained in Y. In fact 


Yio) = YN (i(¥ — a) +a). 


We note that Yj,; depends continuously on (Y,a) for the natural topology on hyperplanes and 
points. 

Conversely, every complex hyperplane Zin C” is contained in a real hyperplane, but there are 
now several choices. If a complex hyperplane Z is given and is defined by the equation 


G-(z—a) =0, then for any complex number @ with |6| = 1 the real hyperplane Z (1 defined by 
Re6(B-(z—a)) = 0 contains Z. The real hyperplane Z!1 does not depend on the choice of 
a € Z and satisfies (24), = Z for every b € Z. 

If a real hyperplane Y and a point a € Y are given, then (Yin) = Y for two values of 0 
with || = 1. Explicitly, if Yis given by the equation Re 6 - (z — a) = 0, then Yjqj is given by 
B-(z—a) =0and (Yiq) by Re (8 - (z — a)) = 0; the two choices 9 = +1 give us Y back. 

For the definition of the real and complex tangent spaces to an open subset 0 of C” with 


boundary of class C', as well as the real and complex tangent planes, we refer to Definition 
9.4.19. 
Clearly Tp,c(b) = To,r(b) (0)3 for the tangent planes, b + Ta,c(b) = (6+ Tar(d)) By: 


Definition 9.7.1 If A is a subset of C", we shall denote by I'4(a) the set of all complex 
hyperplanes Z which pass through the origin and are such thata + Z does not intersect A. 


Definition 9.7.2 A mapping F' : X — YAY) will be called a multifunction from X into Y and 

will be written F : X=3Y. (This means that the value, image, or fiber F(x) of F at a point x is a 
subset of Y, possibly empty.) The graph of a multifunction F, denoted by graph(F'’), is the set 
{(z,y) © X x Y; y€ F(z)}. 


If X and Y are topological spaces, we can equip X x Y with the Cartesian product topology. In 
all cases considered here, X is a T; space—equivalently, all singleton sets are closed. If so, for 


graph(F’) to be a closed subset of X x Y, it is necessary but not sufficient that the fiber 
F(a) = ({a} x Y) Ngraph(F) 


be a closed subset of Y for every a € X. 

Thus I'4 is a multifunction [4 : C’3Gr,_1(C”) = Myrn_i(C) with values in the 
Grassmann manifold of all complex hyperplanes in C” passing through the origin. If 0 is open, 
I9(a) is closed for every a € C”. See also Proposition 9.7.15. 

Lineal convexity of a set A means that I'4(a) is nonempty for every a € C”\ A; weak lineal 
convexity of an open set Q that I"p(b) is nonempty for every b € 02. 

Let us agree to say that a topological space is connected if the only sets which are both open 
and closed are the empty set and the whole space (not necessarily distinct).° A subset of a 
topological space is said to be connected if it is connected as a topological subspace. 

Zelinskii (1981) has proved that a bounded lineally convex open set 0 is C-convex if and only 
if I'g(b) is connected for every boundary point b. See also (Andersson, Passare & Sigurdsson 
2004:46, Theorem 2.5.2) for the corresponding result on subsets of projective space. 


We follow here Bourbaki (1961:1:§11:1) in that the empty space is defined to be connected. Adrien Douady (personal 
communication, 2000 June 26) argued for the empty space not to be connected. The difference is important in Definition 5, where 
C-convexity is defined. 


9.7.3 Accessibility 


We refer to SubSection 9.3.7 for the set-theoretical operations 05, €5, Ks and a, with respect to a 
structuring element S. It is convenient to express closedness and openness for some of these 


operators in terms of accessibility: 


Definition 9.7.3 If A is a subset of R™ or C” and b a point in this space, we shall say that b is S- 
accessible from the outside if b belongs to the closure of ag(CA). In particular we shall speak 
about accessibility from the outside by balls of radius r if S is equal to B<(0,r) or Bz (0,7). 


Remark 9.7.4 If b is S-accessible from the outside of a certain class, then there is also a set T of 
the same class such that AN T = {o}. Indeed, if S satisfies 


{z; f(z) <0} CSc {a} f(x)<0}, 


then T can be taken as the set of all x such that f(x) + || — b||2<0. 


We shall consider regularity classes C’ kB where k € N and 0< B<1, meaning that the functions 
considered are of class C* and all derivatives of order k are Holder continuous of order £, with 
the understanding that C*° = C*, 


Definition 9.7.5 If b € OA is accessible from the outside by a structuring element S having 
boundary of class C® with k>1, then we shall say that the unique tangent plane to S at b is an 
external tangent space of A at b. The set of all external tangent spaces at a point b, a subset of the 
Grassmann manifold Grm—1(R™) = Mmm-—1(R) of all real hyperplanes passing through the 
origin, will be denoted by Ofn (b), and the corresponding multifunction DASGr,_1(R™) by 
elf. 
AR 

If Q is an open subset of C”, we shall denote by en (0) the set of all complex hyperplanes 
through the origin contained in planes in OR (0) ; we call them complex external tangent spaces. 
It is the set of all complex hyperplanes Z = Yi, Y € OBR (0). 


When the class is clear from the context or is unimportant, we shall omit the superscripts ©? 


—~ —Qnw 


It is easy to see that On = Oor = OOR: 
The relation between I"¢(b) and 0%, ,(b), b € OM, seems to be of interest. 


Definition 9.7.6 Let us say that Q is tangentially lineally convex at b © O92 if no complex 
external tangent plane of class C? at b meets 0, i.e., if 0% ¢(b) C T'a(b). 


Proposition 9.7.7 Letb © A C R™ be accessible from the outside by balls of radius r > 0. Then 
e*8 (b) is connected. 


Proof Take b = 0 and assume that A 1 U; = {ol, j = 0,1, where U; is the set of all points x 


such that f;(x) < 0, and f, is a function of a given regularity and with nonvanishing gradient 


wherever it is zero. This is justified by Remark 9.7.4. We now form f, = (1—s)fo + sf1, 
O0<s<1, and claim that the set where f, is negative defines an open set U, which serves to prove 


that all gradients 


(gradf,)(0) = (1— s)(gradfo)(0) + s(gradf,)(0) 


can occur, implying that there is a curve connecting the hyperplane defined by 4 to that defined 
by f,. We note that the gradient of f is nonzero at the origin except in the case when 


(grad f,)(0) is a negative multiple of (grad fo)(0). In that case, however, the hyperplanes 
defined by the two gradients are the same, so there is nothing to prove. We modify f, outside a 
neighborhood of the origin if necessary to make sure that it satisfies the requirement that its 
gradient be nonzero everywhere where the function itself vanishes. 


If xe A\{o}, then x €U;, 7=0,1, so that f;(x) > 0, j7=0,1. This implies that 
f(x) > 0, so that 2 ¢ U,. Thus we have proved that A NU, C {o}; obviously A NU, D {o} 


. In conclusion, we have proved that the tangent plane of U, at b = 0 belongs to e** (0) for all s 
with O<s<l. 


Example 9.7.8 Let us define a cleistomorphism 
Kk, : A(C”") > AC”) ork, : A(R™) > A(R”) 


as the cleistomorphism with structuring element U =CB<(0,r) for some positive radius r. It 
follows that k,(A) is closed for any set A, perhaps most easily seen by observing that its 
complement, denoted by a,(C.A), is the union of all open balls B<(x,r) that are contained inCA 


Thus «,(A) is the smallest invariance set containing A whose boundary points are all 
accessible by balls of radius r, and we see that the boundary points of a closed set F are 
accessible by such balls if and only if k,(F’) = F. 

To treat open sets, we define X,( A) as the interior of k,(.A). In view of Proposition 9.3.8 the 
operation A+ \,(A) = (,(A)) is an ethmomorphism. If we restrict it to open sets, it is larger 
than the identity, i.e., A,(2) D> 2 for all open sets Q. So accessibility for open sets is defined by 
the fixed points of 1, 


The infimum of all the x, r > 0, 1s just the topological closure. 


9.7.4 Concavity and convexity with respect to a structuring element or a family of 
structuring elements 


Just as it is sometimes easier to look at lineally concave sets rather than lineally convex sets, it 
can be more convenient to define accessibility from the inside than from the outside. We shall do 
this in terms of concavity and convexity with respect to a structuring element, treating both 
properties in parallel: 


Definition 9.7.9 Given a subset S (called structuring element) of an abelian group G, we shall 
say that a subset A of G is S-concave if it is a union of translates x + S with x in some subset X 
of G. We shall say that it is S-convex if its complement is S-concave. 

We define the S-kernel of a set A, denoted by ag(A), as the union of all translates x + S 
contained in A. We define the S-hull of a set B, denoted by xs(B), as the complement of the S- 
kernel of CB. 


Obviously Cas(A) = Kg(CA). 
The anoiktomorphism a5 


ag(A) = U(x +S;2+S5C A), Ae AG), 
x 

has as fixed points the S-concave sets. We have ag(A) C A C «gA) for all nonempty sets 

S € AG) and all A. 

We can consider the sets x + S as voxels or pixels, and see that no smaller sets are allowed to 
build up a S-concave set. Or we can think of elements x as atoms and sets z + S as molecules— 
no free atoms are allowed; they must all be part of a molecule. 

What we have done so far is define concavity and convexity with respect to a single set S. Let 
us also consider families .” of structuring sets: 


Definition 9.7.10 Given a family of subsets of an abelian group G, we shall say that a subset 
A of Gis S -concave if it is a union of translates x + S witha € X C G, S € S. We shall say 
that B is S -convex if its complement is Y -concave. 

We define 


ay(A) = Useg (4 + S} 2+ SC A), 
SES 


called the S -kernel of A, and k.y(B) = Cay(CB), called the S -hull of B. 


Thus {.S}-concavity is the same as S-concavity. 

Classical examples are when we take .¥ as the family Y of all open half-spaces in R”, 
defined by an inequality € - x > c, or the the family @ of all closed half-spaces in R™, defined by 
an inequality €-x>c, with € € R™\{0}, c€ R. We can also consider the set of all real or 
complex hyperplanes, or intersections of complex hyperplanes with balls. 


Example 9.7.11 The set A = ]0,1[? U {(0, 0)} C R? (an open square with a vertex added) is 


convex, but is not an intersection of open half planes, nor of closed half planes; in other words, it 
is not evenly convex in the sense of Fenchel (1952)—-see the discussion about this class of sets in 
SubSection 9.2.4, page 257. We obtain 


A = ]0, 1[?CACKa(A) = (0, 1[’Cre(A) = (0, 1)’ = A, 


indicating that more general half planes are needed. 


In view of the above example we now define more general half-spaces, called here refined half- 
spaces, by which we mean convex sets Y such that 
{w#ER”;, €-r<chC Yc {re R”; €-r<c} 


for some € € R™\{0} andc € R. Let us denote by Y the family of all such sets Y. 
Obviously «¢(A) is always a closed set. In view of the Hahn—Banach theorem it is equal to 
the closed convex hull of A. The mapping ky takes an open set to its convex hull (which is open) 


and a compact set to its convex hull (which is closed). 

This is convexity viewed from the outside. We can also work with convexity from the inside: 
We defined in Definition 9.2.7 on page 256 the convex hull of a set A C R”™. It can easily be 
proved that cvxh = ky, showing that the refined half-spaces serve also for convex sets which 
are not evenly convex—see Section 9.2, page 257, for these sets. 

The operation cvhx maps any set to its convex hull, which need not be closed even if A is 
closed. The composition clos*cvxh takes any set to its closed convex hull. (The composition 
cvxh ‘clos is not idempotent if m>2.) 


Definition 9.7.12 We shall say that an open subset of R™ or C” is r-concave if it is a union of 
open balls of radius r. A closed subset is called r-concave if it is a union of closed balls of radius 
r. A Set is called r-convex if its complement is r-concave. 


This definition agrees for open sets in C with that of Sergey Favorov and Leonid Golinskii 
(2015:3). They defined the r-convex hull of a set E C C, denoted by conv(£), as the set 
r 


conv,(£) = N(CD<(z, r); E COD <(z, *)) EcC,r>0. 


Thus Ceonv(£) is a union of open disks. They call a set r-convex if conv(E) = E. Such a set 
r Yi 
is always closed. The generalization to R” or C” is obvious, and we see that conv(£) is exactly 
r 


the set &g_(0,r)(£) with the notation from Definition 9.7.9. When r tends to +00, we get the 


closed convex hull evxh(F) as a limiting case.’ 


7The notion of convex closed sets is used by these authors as an hypothesis in results on Blaschke-type conditions for the Riesz 
measure of a subharmonic function, thus in a context quite different from the one studied here. Since I worked on generalized 
convexity during the period 1996-2001 (see for example Proposition 4.9 in my paper (1996) and then again since 2014, and with 
quite different problems, our respective studies are independent. 


9.7.5 Lineal convexity viewed from mathematical morphology 


Lineal concavity is an example of .¥-concavity, taking .Y equal to the family 2 of all complex 
hyperplanes in C” containing the origin. Weak lineal convexity means that « (2) does not meet 
the boundary of 0. 

There are also local variants of these definitions: we take Y = %, as the family of all 
intersections ZM B<(0,r), where Z is a complex hyperplane passing through the origin. The 
corresponding Z,-convexity, for some positive r, can be called uniform local lineal convexity. 

Let us take again the family .” of structuring elements in Definition 9.7.10 as the set 
¥ Cc A(A(C")) of all complex affine hyperplanes in C” We define a dilation 
~: A(#) > AC") by 


WB@)= UZ, Be AZ). 
(9.75) 


Its lower inverse y_1) : A(C”") + A(Z) is defined by 


b(A) =U (Bi WB) CA)={ZE BZA}, = AE P(C"). 


(9.76) 


We note that € = y_1) is an erosion—as the lower inverse of a dilation, but also easily seen 
directly. There is a relation between I", and ¢: 


I'4(b) = {Z € e(CA); be Z}. 
The upper inverse el] : A(#) > A(C) of «is a dilation defined by 


el-H(B) = ia) > A)= pe = (4), BEA LF). 


(9.77) 
By composition we obtain an anoiktomorphism ay : A(C") > A(C”): 
ag(A) = (eN"e)(A) = (Wd-y)(A) = UZ; ZC A), Ae AC"), 
the union of all complex affine hyperplanes contained in A. We can also form 
no(B) = (eel ")(B) = (WB, Be HZ). 


We have ay(A) = A (equivalently ag(A) > A) if and only if A is lineally concave, which 
happens if and only if CA is lineally convex. If is open, it is lineally convex if and only if 
ag(CQ) > CQ, and weakly lineally convex if and only if ax(CQ) D AN. 


9.7.6 Exterior accessibility of Hartogs domains 


We shall now study Hartogs domains in C” x C, where we write coordinates as 
(z,t) Ee C® x C, 

To define complete Hartogs sets, we may use either the function R, the function h = R?, or 
the function f = —logR. An open complete Hartogs set is then defined equivalently by 
It] < R(z); |t)? < h(z); |t| <e-/, and we are free to choose whichever is convenient for a 
specific calculation. We note that if fis plurisubharmonic, then , defined by log |¢| + f(z) < 0, 
is pseudoconvex. 

Complex hyperplanes in C” x C are of three kinds: 


1. A hyperplane can be given by an equation 8 - (z— z°) =0 for some 8 € C”\{0} and 
some point z° € C” (we shall call it a vertical hyperplane ). 

2. It can have the equation t = c for some complex constant c (we shall call it a horizontal 
hyperplane ). 

3, Finally it can have the equation t = 8: (z — z°), where # is nonzero. Such a hyperplane 
intersects the hyperplane t = 0 in a hyperplane in C” containing 2’. 


The projection C” x C3(z,t) +> (z,|t|)¢ C" xR can be used to visualize the set. 
Equivalently, we can look at the intersection of 0 with the set {(z,t);z¢€C”,t>0}. A 
hyperplane is then represented in C” x R by either 


1. a vertical plane; 
2. a horizontal plane |t| = |c|; or 


3. acone |t| = |8- (z— a )| with vertices at all the points z satisfying 6 - (z — z”) = 0; when 
n = 1 just the unique point 2. 


If b = (z°, t°) is a boundary point with t° = 0, then there is a complex line of equation z = z° in 
the complement of ©, and there may or may not exist a hyperplane in I’9(b) —if the set w in C” 
where R is positive is lineally convex, there is such a hyperplane. If on the other hand 
b = (2°, t°) is a boundary point satisfying |t°| = R(z°) > 0, then a hyperplane Z € I'p(b) is 
given by an equation t/t® = G-z; the parallel hyperplane 6+ Z passing through b has the 
equation t/t? = 1+ 8- (z— 2°). It may happen that all real hyperplanes containing b + Z cuts 
Q, but if this is not the case, the only real hyperplane containing b+ Z and not cutting () is that 


of equation Re (t/t°) = 1+ Ref - (z— 2’). 


Theorem 9.7.13 Let a function R: C" — |—oo,+00] be given and consider the complete 
Hartogs set Q defined as in Definition 9.4.2. Assume that Q is open and weakly lineally convex. 
Then R is continuous at every point where it is finite and positive, and all boundary points of Q 
satisfying (z°,t®) with |t°| = R(z°) > 0 are accessible from the outside of class C?. In fact, 
every complex hyperplane which passes through a boundary point (z°,t°®) with 
\t° | = R(z°) > 0 and does not meet Q 1s contained 1n a real external tangent plane. In particular 
I'9(b) C On,c(b) for all points b = (z°,t°) with |t°| = R(z°) >0 @pc(b) is defined in 
Definition 9.7.5). 


Proof Any point (z°, t°) with |t°| = R(z°) > 0 belongs to the boundary of Q, so there exists by 
hypothesis a vector 8 € C” such that the complex hyperplane defined by t/t? = 1+ f- (z— 2°) 
lies entirely in the complement of 0. We shall prove that there is a real external tangent plane of 
class C’ containing it. 

That the complex hyperplane does not meet 1 means that 


A <|1 + B-(2-2°) 


; zEC”. 


Now 
Ja+z[<$+ Gllt2)?=1+Rez+$|z|?, ze, 
with equality if and only if |1 + z| = 1. It follows that for any y > $, 


1+ 2\<1+Rez+4+4|z/?, zeC, 
| y 


with equality only when z = 0. Hence 


t+ 8-(z—2°)| <1+ReB-(z-2)+4|8- (2-2)? 
<1+Ref-(z—2°) + Ill} - lle — 2°13, 


with equality between the first and last expression only when z = z° or 6 = 0. Therefore, if we 
choose c > +||]|3, 


R(z)/|t°|<1+ReB-(z— 2°) +ellz—2°|)3, ze C*, 


with equality only when z = 2°. 


So the set 
U= { (z, t); Re (t/t) > 1+ReB- (z— 2°) +el|z— 2°|[2}, 


taking c > 5 I6ll. is a set with smooth boundary and the real hyperplane defined by 
Ret/t® = 1+ Re: (z— z°) is an external tangent plane of class C* of (at (z°, t°). 

From what we just proved it follows in particular that R is upper semicontinuous where 
positive. On the other hand, 0 is open by hypothesis, which, as we noted, implies that the 
restriction R|,, is lower semicontinuous. 


9.7.7 Unions of increasing squences of domains 


If an increasing family (V;) jen Of open sets in R” is given with union V and if b € OV, let us 
denote by limsup Oy,n(b), understood as (limsup Oy, R)(b), all limits of real hyperplanes 
Y; € Oy, r(b™) at points b) € AV; such that 6”) — b as j > oo. Here Oy(b) is defined in 
Definition 9.7.5. We shall use a similar notation for the complex hyperplanes: lim sup O 2,,c(b) 
when £2; increases to Q, and also lim sup Ig, (b). 


Proposition 9.7.14 Let (V;) jen De an increasing family of open subsets of R™. Define Oy p as 


in Definition 9.7.5 using as structuring element a set S with boundary of class C®’ with k>1. 
Then Oy n(b) C limsup Oy,.R(b) for all points b € OV. A similar result holds for the complex 
external tangent planes © ¢ ¢(b) of an open subset Q of C”. Here the inclusion can be strict. The 
limit superior is always nonempty. 


Proof Take b = 0 and let U be an open set with boundary of the class in question such that 
VnU= {ol, defined as the set of all points x where v(x) is negative, y being of the right 


class and with nonvanishing gradient where it is zero. Let g,, s > 0, be the function 
p(t) = 9(«)—s +23, eR", 


and let U, be the set where 9g, is negative. We note that when x € V, then x ¢ U, so that y(x)>0 


If x € VOU,g, then v(x) >0 while v(x) < 0. So ||x||? < s — y(x)<s. Since @ is of class C’, 
its gradient at any point in V MU, is close to its gradient at the origin. For every large enough j 
there is a smallest s; such that U,, and V; have a common boundary point b'). Necessarily, then, 


b(9) ||2<s. For large j, sis small, so small that the external tangent plane of U,. at b is as close 
[| Il ge j, 5; gent p ; 


as we like to the tangent plane of U at the origin. This shows that any hyperplane in Oy R(0) can 
be approximated by hyperplanes in Ov,n(b") J; 


Proposition 9.7.15 Let (225), en De an increasing family of lineally convex open subsets of C” 


and denote their union by Q. Then limsupI9,(b) = I'n(6). In particular the graph of Ig is 
closed. 


Proof If Z ¢ In(b), then b + Z intersects 0. Take a compact ball K in ( that contains a point of 
b+ Z in its interior. Then for all suffiently large j, 92; contains K. All hyperplanes which are 
close enough to 6 + Z intersect K and hence also {2; for these j. Therefore, if Z; tends to Z and 
b9) € O02; tends to b, then bi) 4 Z; intersects (2; for large j. This means that hyperplanes 
b) + Z; with Z; € T9,(6) cannot approach b + Z. So we have lim sup I'n,(b) C Ia (0). 

The opposite inclusion is trivially true. 


Lemma 9.7.16 If A is a closed set inR™ and b € OA, then ©?, ,(b), where we use a Euclidean 
ball as structuring element, is nonempty. 


Proof Given b € OA and a positive number s, take c ¢ A with ||c — b||, < s. Take then r > 0 
maximal so that B-(c,r) does not cut A. Clearly r<s. On the boundary of this ball, there must 
exist a point p € A. Then ||p — b||2<r + s<2s, and p is accessible from the outside of class C, 
which means that OrR (p) is nonempty. Since s is arbitrarily small, the closure of the the graph 
of On~R has a nonempty fiber over b. 


Theorem 9.7.17 Let Q be an open subset of C" such that it is equal to the interior of its closure. 
If Q is tangentially lineally convex at all points b in some open subset B of 092 (see Definition 


9.7.6), then OF &(b) C 62, Q(b) C I'a(b), and I'9(b) is nonempty for all b € B. In particular, 
tangential lineal convexity at all points b € O92 implies weak lineal convexity. 


Proof We apply Lemma 9.7.16 to A =. Then the interior of A is equal to ©. Moreover, 
graph(I',) is closed; see Proposition 9.7.15. 


If Q is lineally convex, I’9(b) is not necessarily connected, not even when () is a Hartogs set, as 
is shown by the example below as well as by Example 9.4.8 in Section 9.4, page 281. 
Example 9.7.18 Let Q be the Cartesian product of an annulus and a disk, 


Q= {(z1, 22) E Our 1< |21| < 2, |Z2| < 1}, 


a lineally convex set. We define complex hyperplanes Zg passing through O by the equations 


Bz, = (1— B)zo, z € C”, BE [0,1]. Use a ball Be (0, r) withO <r < 1 as structuring element. 
Then @%, .,(b), where b = (1,1), consists of all the Z, 8 € [0,1], whereas (b) consists of Zo 


and Z, for $<B<l. Thus I'9(b) does not contain O74 lO). We also note that I'Q(b) is not 
connected; it has two components, {Zo} and {Z B} + <B<l}. 


If0<B< +: then there are points z = (—1— 8,22) € Zg 2 far from b = (1,1) (take 
s>0,s=—-1—- 2% < 2/8 — 4) as well as points in Zg N 22 arbitrarily close to b. 

The set Q is lineally convex, but if we approximate it from the inside by a set with boundary 
of class C' containing all points in Q with distance to 022 at least equal to c > 0, then we get a 
set which is 2\-convex but not Z,-convex for r>1+ «> 1. (For S-convexity, se Definition 
9.7.10; for Z,-convexity, see the beginning of SubSection 9.7.5.) 


9.7.8 Convexity properties of superlevel sets 


Definition 9.7.19 Given any function f on a set X and with values in the set R. of extended real 
numbers and an element c of |—oo,+00], we define its (non-strict) supetlevel set as 
{a € X; f(x)>c}. Analogously we define its (non-strict) sublevel set as {x € X; f(x)<c}. 


Given a complete Hartogs set with radius function R, we shall denote by M, the superlevel set 
{z€ C0"; R(z)2c}. 


Example 9.7.20 Consider the lineally convex Hartogs set 22 Cc C x C defined by the radius 
function 


R(z) = min(|z — 2|,|z+2|),  |z| <1; Riz) =0; |z<. 


Then (0, 2) belongs to the boundary of Q and I'g((0,2)) consists of precisely two elements, the 
hyperplanes defined byt = —z andt = z, respectively; thus it is not connected. (This shows that 
Q is not C-convex in view of Zelinskii's criterion mentioned near the end of SubSection 9.7.2.) 
However, the union of all the 9(b) with b € 02 is connected. We note that P'p((i, W5)) is 
connected and contains '9((0, 2)). See Example 9.4.8 in Section 9.4. 

The boundary points of Q are accessible from the outside by balls of a not too large radius, 
and O,c((0,2)) consists of all hyperplanes t = Xz, with X € [—1,1]. We also note that the 
intersection of Q with the complex line t = c has two components if 2< le| < V5. 

Also, for a = s +i(1—s/2) with a small positive number s, the superlevel set M R(a) 18 
B.(0,r)-convex for r slightly smaller than V5, whereas for s = 0, a =i, the superlevel set 
Mri), now equal to {i, —i}, is B<(0,1r)-convex for any r but not convex. (This is a warning that 
r-convexity is not so meaningful for sets that are not regular open or regular closed.) 


For simplicity we shall assume below that n = 1. 


Theorem 9.7.21 Let (£2 Cc C x C be a complete lineally convex Hartogs domain defined as in 
(9.42) with n = 1. Assume that a point a € w is such that R(a) < sup R. Then there exists an 
r > 0 such that if a belongs to the erosion €p_(o,)(w), then a € ap -(0,r) (CM aay): In other 
words, since a belongs to the open set w, the distance r to Cw is positive, and a is exterior 
accessible in Mpyq) by disks of radius r. 


Proof There is a complex hyperplane (thus a complex line in the present situation) in the 
complement of 0 which passes through (a, R(a)). It cannot be vertical since a € w and it cannot 
be horizontal since R(a) < sup R, so it must have an equation of the form 


Rta = 1+ B(z-a) = A(z — az), 


where 8 # 0 and ag = a — 1/{ is the point where the line hits the line t = 0. 

This implies that the cone in C x R defined by |¢|/R(a)>|8(z— ag)| does not meet any 
point (z, |t|) € 2, in particular that a belongs to the disk D<(ag, s) with center at ag and radius 
s = |a—ag| = 1/|A|. As noted, this disk does not meet @, so a € ap_(o,s)(CMp(q)). We note 
finally that s = |a — ag|>d(a, Cw) = r. 


There is no uniformity here: r depends on a. But if 0 is bounded and we restrict attention to 
points a in a compact subset of w and with R(a)>c > 0, we can choose a fixed r > 0. Thus 
Mrva) is D<(0,r)-convex. 

There may be several lines of the form t = 6(z — ag) as mentioned in the proof. Then among 
all the possible values of @ € C we can take the infimum of their absolute values, and any limit 
of these numbers must also define a line in the complement of (, since the complement is closed. 
This gives the largest possible value to r = 1/|]. 

In Example 9.7.20 we see that, for a real such that 0 < a < 1, 


r=2-a>d(a,lw) =1-a, 
implying that the number r obtained in the proof can be smaller than it is in an actual situation. 


Remark 9.7.22 In the other direction, if a closed r-convex set M in C is given, then there exists a 
lineally convex open set in C" x C with radius function R such that Msupr = M; see 
Proposition 9.4.12 on page 284. 


Corollary 9.7.23 If 0 is lineally convex and bounded, and its boundary is of class C’' at the set 
where R > 0, then a pointa € w belongs to ap-(0,r) (CM ria) if 
1 
"Sereda 
This is the case for all points z with R(z) = R(a) if 


1 
"S Spl ered Re) R@)=RO)) 


We see that r 7 +00 when R(a) /“ sup R, meaning that the superlevel set becomes more and 
more convex. We shall make this precise in Theorem 9.7.29. 


Proof In this situation there is only one line in the complement of 0 passing through (a, R(a)), 
and the absolute value of the coefficient f is ||(gradR)(a)||2 = 2|R,|. The radius rdepends on a 
and may vary, but among all the points z with R(z) = R(a) its lower bound is positive. 

We now consider a situation with two levels, R(a) and R(a) + s>R(a). 


Theorem 9.7.24 Let 2 C C x C be a lineally convex Hartogs domain defined as in Definition 
9.4.2 with n = 1 and take a pointa € w C C with R(a) < sup R. Then there exists a number 
r > 0 such that for all s>0, 


d(a, MRa)+s)2 Ra’ 
where the inequality means that any point w with R(w) = R(a)+ s is outside the disk 
D<(ag,11) withr; = r+ sr/R(a). 

It follows that w is accessible with disks of radius r; in the complement of the superlevel set 
MR«\a)+s- 
Proof As in the proof of Theorem 9.7.21, we see that the cone defined by |¢|/R(a)>|8(z — ag)|, 
where f/ is the coefficient in the equation of the line in the complement of 0 passing through 
(a, R(a)), viz. t/R(a) = 1+ B(z — a), does not contain any point of the form (z, |¢|) in Q. We 
take r=1/|6|. In particular the disk D<(ag,r1) with r; =r+sr/R(a) for any w with 
R(w) = R(a) + s does not meet Mziq)+-5- 

Since also (w, R(w)) admits a line t/R(w) = 1+ y(z—w) in the complement of 0, we 
must have |7|<|6], so the corresponding radius rz = 1/|+¥| is not smaller than ry. 


9.7.9 Admissible multifunctions 


Definition 9.7.25 Let Q be an open subset of C" and y: B3Gr,_1(C") a multifunction 
defined on a subset B of the boundary of Q and with values in the Grassmann manifold of all 
hyperplanes through the origin. Consider the following three conditions on y. 


¥(b) C Pg(b) forallb € B; 


(9.78) 
thegraphof yisclosed; and 
(9.79) 
7(b) isnonemptyandconnected foreveryb € B. 
(9.80) 


We shall say that y is admissible if these conditions are satisfied. 


It follows that graph(y) is connected if 0.2 is connected; see Lemma 9.7.27 below. 

An example of an admissible multifunction is [9 : 0Q=3Gr,_1(C") provided I9(b) is 
connected for every b € OM. (In particular, this is the case if the boundary is of class C!.) The 
graph is then automatically closed in view of Proposition 9.7.15. It is easy to see that in 
Examples 9.7.18 and 9.7.20, there is no admissible multifunction y in any neighborhood of the 
points (1, 1) and (0, 2), respectively. 

If © is tangentially lineally convex, a candidate for y might be the closure of Ogc¢. Then 
property (9.78) holds by hypothesis, (9.79) by construction, and (9.80) may hold if the boundary 
of 0 is sufficiently regular. 


Example 9.7.26 Let Q be a convex open set in C”. If Q is empty or equal to the whole space, 
then its boundary is empty. If Q is a slice, then its boundary has two components. 


In all other cases, 02 is connected, and we know that the set of all real hyperplanes passing 
through a fixed boundary point b and not intersecting Q is connected. Then also the set of all 
complex hyperplanes containing b and contained in such a real hyperplane is connected—the 
mapping Y ++ Yj is continuous as we noted in Section 9.7.2. Thus Ig is an admissible 
multifunction except in the first-mentioned cases, even if the boundary is not of class C?. 


If a lineally convex open set has a C! boundary, I"p(b), a singleton set, depends 
continuously on b. When I’p(b) is no longer a singleton, the following result will serve instead of 
the continuity. 


Lemma 9.7.27 Let Q be an open set inC” andy: A3Gr,,_1(C”) an admissible multifunction 
on a subset A of O22. Then the graph of y over B, 


graphz(y) = {(6,Z); b < BandZ € 7(b)}, 


is connected for every connected subset B of A. In particular the graph of y is connected if the 
boundary of Q is connected and y is defined on all of it. 


are disjoint and closed relative to 


Proof Assume that graph(y) = Vo UVi, where the V;, 
B 


graph(7). Define B; as the set of all points b such that some hyperplane in (B) belongs to V;, 
B 


j = 0,1. Then Bp and B, are disjoint, since by hypothesis every y(b) is connected. Moreover By 
and B, are closed relative to B, since the graph of y is closed and the manifold Gr,_1(C”) is 
compact. By hypothesis B is connected, so either By or B, must be empty. Hence Vo or V, is 
empty, proving that the graph of y over B is connected. 


We note that *(B) is connected as a continuous image of the graph (it is the projection of 
the graph on the target space Gr,,_;(C”)). 


Proposition 9.7.28 Let 2 be an open subset of C” and F an affine subspace of C". Denote by 
QQ the set 21 F considered as an open subset of F. Every complex hyperplane Z in C” which 
does not contain F gives rise to a complex hyperplane y(Z) = ZF in F. Let an admissible 
multifunction y: B3Gr,-1(C”) be given and define a multifunction yp on BN OQr by 


yr(b) = {u(Z); Z € ¥(b)}. Then y, is an admissible multifunction on BN 02 p. 


Proof Since #(Z) C Z, it is clear that yr(b) C Ig, (b); thus (9.78) in Definition 9.7.25 holds. 
The graph of y over any compact subset of O92 is compact; hence the graph of y, over any 
compact subset of Or is compact, thus closed: property (9.79) holds. Finally (9.80) follows 
since y is continuous and thus maps connected subsets onto connected subsets. 


The proposition can in particular be applied to Ig if 'g(b) is connected for all b € 02. 
9.7.10 Links to ordinary convexity 


Theorem 9.7.29 Let R be a continuous real-valued function defined on C” and define Q as in 
Definition 9.4.2. Assume that Q is connected and that its boundary is of class C} (at least in a 
neighborhood of Mgup R). Then the set Mgyp R where R attains its maximum, 


Msp r = {2 € C"; R(z) = sup R}, 
(9.81) 


1S convex. 


Proof A set is convex if and only if its intersection with every one-dimensional complex affine 
subspace is convex. Therefore it is enough to prove the theorem for n = 1. 

So let n = 1 and let a belong to the boundary of Msup r. We shall prove that there is an open 
half plane with a on its boundary which does not meet Mgup r, proving the convexity of that set. 

We have (gradR)(a) = 0, and near a there are points c with (gradR)(c) nonzero and 
arbitrarily small. In view of Corollary 9.7.23 this means that there is a disk of arbitrarily large 
radius with c on its boundary. The disk is of the form D<(cg,r), where r= |c— cgl, 
cg =c—1/f being the point where the line t/R(c) = 1+ 6(z —c) hits the plane t = 0. The 
normalized vectors (c — cg)/|c — ¢g| have an accumulation point, and this proves that the union 
of all the disks D<(cg,r) when c varies in an arbitrarily small neighborhood of a contains an 
open half plane with a on its boundary. We are done. 


The assumption that the boundary be of class C! can be weakened, as we shall now show. 


Theorem 9.7.30 Let R be a continuous real-valued function defined on C” and define Q by 
(9.42). Assume that Q is bounded and connected and that there exists an admissible multifunction 
y defined at all points (z°,t°) with \t°| = R(z°) > 0, thus on the boundary over the base of Q 
(see Definition 9.7.25). Then the set Mgyp R where R attains its maximum 1s convex. 


For a Hartogs domain Q we always have Ip(b) C Oac(b) when b= (z°,t°) with 
\t°| = R(z°) > 0 (Theorem 9.7.13); if the domain is tangentially lineally convex, we have 
I'p(b) = On,c(b). For such domains we therefore have an admissible multifunction 
y = In = Onc: (9.78) is obvious; (9.79) follows from Proposition 9.7.15; (9.80) follows from 
Proposition 9.7.7. 

We note that the hypothesis is satisfied in particular if © is lineally convex and R is of class 
C'. In (Kiselman 1996, Theorem 4.8) the result was proved under this hypothesis, and even under 
the weaker one that R can be approximated from below by C! functions. 

In view of Zelinskii's characterization of C-convex sets mentioned near the end of Section 
9.7.2, the hypotheses are satisfied for C-convex sets, again taking y = I'p. There are easy 
examples which show that Ms,)R need not be convex if we drop the hypothesis of 
connectedness; see Example 9.4.8. 


Proof of Theorem 9.7.30. Again, the set Msupr 1s convex if its intersection with every one- 
dimensional complex affine subspace is convex. Proposition 9.7.28 shows that if we have an 
admissible multifunction on a subset of 02, then there is one also on a corresponding subset of 
OQ, F being any affine subspace of C” x C. Therefore, taking Fas the Cartesian product of a 
complex line in C” and the line z = 0, we see that it is enough to prove the theorem for n = 1. 

So let n = 1. To prove that Mgupr is convex means to prove that the segment [so, s;| is 
contained in Mgupy p if 89, $1 © Mgup pr. There is no loss in generality if we assume that s) = —1 
and s; = 1. 

A non-vertical and non-horizontal complex line through (a, t°) with t° 4 0 has the equation 


4 =1+4+ A(z- a) = B(z— ag), zEC, 
where ag = a — 1/( is the point where the line hits the plane t = 0. We define 


—1/Bif B £0, 
aa, 8) = ern 


In case R is differentiable at the point a, f is uniquely determined if we require that the line be in 
To((a,t°)). 

We denote as before by @ the set of all points z € C such that R(z) > 0. In general the 
external tangent is not unique and we shall denote by Q(a) the set of all points a — 1/8 that can 
be obtained from complex lines in ((a, t°)), thus 


Q(a) = {a(a, 8); BE Ya, R(a))} CS? =CUf{oo}, aew. 
(9.82) 
We define Q(a) = {a} when a ¢ w. Thus Qis a multifunction, Q : S?=3S?\w; its images Q(a) 


are compact and connected. 
The radius can always be estimated by 


R(z)<R(a)|A| - |z — agl, z€C,acw, BE x(a, R(a)), ag = g(a, f), 
with equality for z = a, assuming 6 # 0. In particular, if w © Msup r, then 
R(a)|B| - |a — g(a, B)| = R(a)<R(w)<R(a)|B) - |w — g(a, B)]. 
If ag € Q(a)\{oo}, then necessarily 8 4 0, so that 
la — ag|<|w — agl, a cw, we Mapp, ag € Q(a)\{oo}. 
(9.83) 


Assume that —1 and 1 belong to Mgupr; we shall then prove that any point c € [—1, 1] 
belongs to Mgupr. Consider Q(c+ iy) for real y. We know from Lemma 9.7.27 that the set 
Q+(c + iR) is connected. If w is bounded and y or —y is very large, then Q(c + iy) = {c + iy}. 
In general we can prove that Ima > 1 implies that Imb > 0 for all b € Q(a), and similarly 
Ima < —1 implies Im b < 0 for all b € Q(a). This follows from the following lemma. 


Lemma 9.7.31 If Q is a complete Hartogs domain in C* with radius function R and if 
+1 € Moup pr, then for allb € C with |Rea|<1 and allb € Q(a)\{co} we have 


Im a>limplies Im b> $ (Ima — 1) and 
Im ax — limplies Im b< + (Ima + 1). 


Proof We know from (9.83) that |a — b|<| + 1 — b|. Expanding |+1 — b|? — |a — b|?>0, we get 


2(Reb)(Rea + 1) + 1— (Ima)(Ima — 2Imb)>(Rea)?>0, 


from which we deduce that 1>(Ima)(Ima — 2Imb}), an inequality which implies those in the 
lemma. 


Proof of Theorem 9.7.30, cont’d. So Q(c + iy) must pass from the upper half plane to the lower 
half plane when y goes from large positive values to large negative values, c being fixed. But it 
can never pass the real axis at points with e>1 or x< — 1. Indeed, if bis real and larger than or 
equal to 1, we get from (9.83), taking a = c + iy, 


la —b|<|1—b| =b-1, 


implying Re a>1, so that c>1 contrary to assumption. Likewise, Q(c + iy) cannot pass the real 
axis at a point with e< — 1. 

However, Q(c + iy) cannot pass from numbers with arbitrarily large positive imaginary part 
to numbers with large negative imaginary part in the strip —1 < Rez < 1 either. In fact, w is 
connected, so there exists a curve contained in w connecting —1 to 1, and Q(c + iy) cannot cross 
that curve. 

Hence it is impossible for Q(c + iy) to pass from the upper half plane to the lower half plane 
if it has only finite values. So it must have an infinite value, which means that c + iy? € Moup R 
for at least one y’. 

We thus know that there is a y’ such that c+ iy® € Msupr; without loss of generality we 
may assume that it is nonnegative. Choose y’ as small as possible. If y° = 0 we are done: 
c © Mou r. Let us assume that y? > 0 and try to reach a contradiction. 

By (9.83) any point b € Q(a)\{co} must lie in each of the three half planes 


ja—a|<|1—b]  Ja—BJ<|-1-d], — Ja—B|x<e-+iy? —d). 


The intersection of these three half planes is a triangle, and the union of these triangles when 
a=c+iywithy € [S$ y°, y°] is bounded. Thus the possible finite values for b when a varies as 
indicated is bounded, and for a = c + iy with syosy < y® the point b cannot be infinity. On the 
other hand, when a = c+ iy? € Mgup pr, then Q(a) must contain oo, This means that the set of all 
points b € Q(a) originating from points a =c+iy with y € [sy 9" consists of © and a 
nonempty bounded set; it is not connected, in contradiction to Lemma 9.7.27. This contradiction 
shows that we must have c € Mgup r and proves the theorem. 


It is easy to modify Theorem 9.7.30 using Mobius mappings, at least if n = 1. In fact, any 
mapping 


C x C3(z,t) 4 (SH, ae) = (2) Cx C 


preserves lineal convexity, as was shown in (Kiselman 1996:Lemma 8.1). Denote by a, the point 
where a line t/t® = 1 + A(z — 2°) intersects the z-plane. The line can be mapped by a Mobius 
mapping to a line t’ = constant. This mapping takes the point a, to infinity, and all circles in the 
z-plane which pass through ag are mapped onto straight lines. Convex sets are transformed 
accordingly: 


Definition 9.7.32 Let b be a complex number or ~. Let us say that a subset A of the Riemann 
sphere C U {00} is b-convex if 

(10.4.1). b € A; and 

(10.4.2). :p+(A) is convex if g is a Mébius mapping which maps b to infinity. 


Corollary 9.7.33 (to Theorem 9.7.30) Let Q and y be as in Theorem 9.7.30, assume that n = 1 
and let m denote the projection defined by x(z,t) = z. Consider a line Z € y(a), where 
a = (z°,t°), |t°| = R(z°) > 0, and let b be the point such that (b,0) € a+ Z. Then the set 
m*((a+ Z)M 2) is b-convex. 


ee 
9.8 Duality of Functions Defined in Lineally Convex Sets 


Abstract of this section 

The term duality represents a collection of ideas where two sets of mathematical objects confront 
each other. A most successful duality is that between the space Y(2) of test functions (smooth 
functions of compact support) and its dual Z’(2) of distributions. 

Similarly, the theory of analytic functionals, developed by André Martineau in his doctoral 
thesis (1963)—also in (Guvre de André Martineau 1977:47—210)—is based on a duality, now 
between the Fréchet space of holomorphic functions @({2) in an open set 0 and its dual G’(2). 
In many, but not all respects, it is analogous to distribution theory. 

In complex geometry, lineal concavity and lineal convexity can be treated successfully using 
concepts of duality. 


9.8.1 Introduction to this section 


Lineal convexity, a kind of complex convexity intermediate between usual convexity and 
pseudoconvexity, appears naturally in the study of Fantappié transforms of analytic functionals. 
A set is called lineally convex if its complement is a union of complex hyperplanes. This 
property can be most conveniently defined in terms of the notion of dual complement: the dual 
complement of a set in C” is the set of all hyperplanes that do not intersect the set. It is natural to 
add a hyperplane at infinity and consider C” as an open subset of P”, complex projective space of 
dimension n. The definition of dual complement is then the same, and somewhat more natural: 
the set of all hyperplanes is again a projective space. In this setting, the dual complement is often 
called the projective complement. Indeed, Martineau (1966) called it le complémentaire projectif; 
the term dual complement used here was introduced by Andersson, Passare and Sigurdsson in a 
preprint from 1991 of their forthcoming book (2004). 

We can now simply define a lineally convex set as a set which is the dual complement of its 
dual complement (here it becomes obvious that we should identify the hyperplanes in the space 
of all hyperplanes with the points in the original space). So this duality works well for sets. What 
about functions? 

In convexity theory, a convenient dual object of a set is its support function as defined in 
Section 9.2. For functions, we have the Fenchel transformation, defined as well in Section 9.2. 

Is there a duality for functions that generalizes the duality for sets defined by the dual 
complement? In this section we shall study such a duality. We call it the logarithmic 


transformation. It has many properties in common with the Fenchel transformation. However, 
there are some striking differences. The effective domain, defined by formula (Definition 9.2.5), 
of a Fenchel transform is always convex, but the effective domain of a logarithmic transform 
need not be lineally convex (Example 9.8.16). This is connected with the fact that the union of an 
increasing sequence of lineally convex sets is not necessarily lineally convex (Example 9.8.17). 
However, the interior of the effective domain of a logarithmic transform is always lineally convex 
(Theorem 9.8.14), and the transform is plurisubharmonic there (Theorem 9.8.18). 

Working with functions defined on P” is the same as working with functions defined on 
Chr io} which are constant on complex lines, i.e., homogeneous of degree zero. For instance 
a plurisubharmonic function on an open subset of P” can be pulled back to an open cone in 
Cres 0} and the pullback is plurisubharmonic for the 1+ coordinates there. However, I 
cannot define a duality for such functions. I have been led to consider instead functions defined 
on subsets of ci\ {o} which are homogeneous in another sense: they satisfy 
f(tz) = — log |t| + f(z). Such functions are not pullbacks of functions on projective space, but 
the duality works for them. In a coordinate patch like z) = 1 we can identify them with functions 
on a subset of P”. Given any function F on C”, we can define a function f on c'*™\ {o} by 
f(z) = F(21/20,---;2n/Z0) + clog |zo| when zo 4 0 and f(z) = +00 when zp = 0, where cis 
an arbitrary real constant; this function is homogeneous in the sense that f(tz) = clog |t| + f(z) 
, SO we can choose any type of homogeneity. In other words, locally all kinds of homogeneity are 
equivalent, and there is no restriction in imposing the homogeneity we have here, viz.c = —1. 

As mentioned in Section 9.2, there are several other notions related to lineal convexity. The 
property called Planarkonvexitét in German (see Behnke & Peschl 1935), or weak lineal 
convexity is weaker than lineal convexity: an open connected set is called weakly lineally convex 
if through any boundary point there passes a complex hyperplane which does not intersect the 
set. Aizenberg (1967) proved that these domains are precisely the components of Q”™ (for 
notation see SubSection 9.8.2 below). 

Strong lineal convexity was defined by Martineau (1966:Definition 2.2) as a topological 
property of the space of holomorphic functions in a domain. Martineau (1966: Theorem 2.2) and 
Aizenberg (1966) proved independently that convex sets are strongly lineally convex. The 
property was given a geometric characterization by Znamensky (1979). This geometric property 
is now called C-convexity. Its relation to lineal convexity has been studied by Zelinskij (1988) 
and others. For these two properties we refer also to the survey by Andersson, Passare and 
Sigurdsson (2004) and the monograph by Hérmander (1994). 

Another generalization is the notion of m-lineal convexity to be studied in the next section. 


9.8.2 Notation 


Let A be a subset of C'*"\ {0}, where n>1. We shall say that A is homogeneous if tz € A as 
soon as z € A and t € C\{0}. To any homogeneous subset A of chor we define its dual 


complement A’ as the set of all hyperplanes passing through the origin which do not intersect A. 
Since any such hyperplane has an equation €-z=Coz%o +-::+n2, =0 for some 
CE cen {0}. we can define 


A’ = {¢€ CM\{0}; ¢-z # Oforeveryz € A}. 


(9.84) 


Strictly speaking, we should have two copies of cln\ {o} (a Greek and a Latin one), and 
consider A’ as a subset of the dual (i.e., the Greek) space. A homogeneous set is called Jineally 
convex if C1*"\A is a union of complex hyperplanes passing through the origin. A dual 
complement A’ is always lineally convex, and we always have A™ > A. The set A” is called 
the lineally convex hull of A. A set A is lineally convex if and only if A= A>. 

The operation of taking the dual complement is an example of a Galois correspondence, and 
the operation of taking the lineally convex hull defines a cleistomorphism in the ordered set of all 
subsets of cl\ {fo}. For the general definitions of these concepts, see Section 9.3. 

We shall write z = (zo, 2’) = (z0, 21,---,2n) for points in crn tot, with z) € C and 
z' = (z1,.--;2n) € C”. Homogeneous sets in c'\ {0} correspond to subsets of projective n- 
space P”, and we can transfer the notions of dual complement and lineal convexity to P”. In the 


open set where zp # 0 we can use z as coordinates in P”. 
We shall denote by 


Ye ={zec'\{0};¢-z=0}, ¢ec'\{o}, 


(9.85) 
the hyperplane defined by ¢. Then the dual complement can be conveniently defined as 
A = ie YynNA=O}, 
(9.86) 
and its set-theoretical complement in c'*"\ {o} is 
[AY = (Cl*"\ {0})\ A" = ve YNAF O}. 
(9.87) 


The complement of the lineally convex hull A” can be written as 


CA“ = UY,. 


ac A* 


We shall use this idea in the following lemma. 
Lemma 9.8.1 For any subset I of C'*"\ {0} we define 
A = C(UyerY,) = Myer ly. 
Then A is lineally convex. Moreover AW = A=I"° andA =I DT. 


Proof Clearly A as the complement of a union of hyperplanes is lineally convex, so AV =A. 
The statement a € A is equivalent to y-a #0 for all y € I’, which by definition means that 


a é€I*;thus A=TI". As a consequence, A =I”. 


How does the operation of taking the dual complement intertwine with the topological operations 


of taking the interior and closure? The answer is the following (we write AU for the interior and 
A for the closure of a set A). 


Proposition 9.8.2 For any homogeneous subset A of C Aral Ot we have 


(A) IfA is open, then A’ is closed. 

(B) IfA is closed, then A’ is open. 

© APcA™. 

(D) A* =A", 

(E) IfA is the closure of an open set, then A’ is the interior of a closed set. 


Proof (A) and (B). To see that A’ is closed if A is open we only have to look at (9.86). The same 
formula shows that A* is open if A is closed. 

(C). Since AU is open, A” is closed according to (A), so that A” = Aton. 

(D). Since A” is open according to (B), and since A “Cc A’, we get A CA. Te prove the 
inclusion A” Cc A” we argue as follows. If ¢ € A’ , then Yy M A = 0 for all @ near ¢ The union 


of these hyperplanes Yp is a neighborhood of Y;, so ¢ € A’. 


(E). If A = B with B open, then according to (D), A* =B* = B’ , the interior of the closed 
set B’. 
This proves the proposition. 


If A is the interior of a closed set C, we get A. = CD Cc possibly strictly. 


Corollary 9.8.3 If a subset A of cl*\ {o} is strongly contained in a set B in the sense that 
A CB, then B’ is strongly contained in A’. 


Proof Using (C) and (D) in Proposition 9.8.2 we _ see _ that ACB implies 
BC B* cA =A". 


Corollary 9.8.4 Ifa subset A of cl*\ {o} is lineally convex, then its interior A is also lineally 
convex. 


Proof If A = B", then A = B™ = B" by (D) in Proposition 9.8.2, which shows that AU is 
lineally convex. 


By way of contrast, the closure of a lineally convex set is not necessarily lineally convex if n>2. 
It turns out that the lineal convexity of the closure is connected with the question whether we 
have equality in (C) in Proposition 9.8.2, as shown by the following result. 


Corollary 9.8.5 Let B be any lineally convex subset of cin\ fo}. Then its closure B is lineally 
convex if and only if its dual complement A = B" satisfies (C) in Proposition 9.8.2 with 
equality. 


Proof Using the lineal convexity of B and then (C) and (D) in Proposition 9.8.2, we get 
Bak CA S22 


Thus equality in (C) is equivalent to B being lineally convex. 


The inclusion (C) in Proposition 9.8.2 can be strict simply for dimensionality reasons. This will 
be clear from the following result, where we use the interior relative to a subspace instead of the 
interior with respect to the whole space. 


Proposition 9.8.6 Let A be a homogeneous set in Cci\ {o} which is contained in a complex 
subspace F of C1*". Let Ay = relint(A) denote the relative interior of A, i.e., the interior 


taken with respect to F Then A* C (Ap) UF®©, where F° is the set 
Bo = fe 5, ab POPE. 


If A is open in F, then A* UF° is closed. 


Note that when F = C!*”, then F° is empty and we are reduced to Proposition 9.8.2. 


Proof Take a point ¢ ¢ (A ey U F©. Then there is a point a € Ar Y¢ and a non-zero vector 
b € F\Y¢. If @ is close to ¢ then the hyperplane Y, cuts the complex line {a + tb; t € C} ina 
unique point a(9,t) close to a, and since a is in the relative interior of A, a(8,t) belongs to A as 
soon as @ is close enough to ¢. Therefore 9 ¢ A” for all these 0, which means that ¢ ¢ re 

Finally, if A is open in F then A* UFO = A* UFO C (Ap) UF? = A* UF®, since 
Apr = Aand F° is closed. 


Example 9.8.7 It is now obvious that the inclusion in (C) can be strict. Take a nonempty 
relatively open set ACF#C™". Then A =0, and A’ =C™\{O}. But 
A* c (Ar) UFO = ASU FS £0 \ {0}. 


Example 9.8.8 Also the inclusion in Proposition 9.8.6 can be strict. There are sets A such that 
A=0, A°=BA40, and B=A’. Thus A’=C™\{0} and 
A =B=B #4 crm fol. Such a set is the set A of all z € C1*? with |z,|? + |z2|? < |zo|? 
and either z; 1s a complex rational or z2 = 0. (Here the only choice for F is the whole space, so 
that F© is empty.) 


Example 9.8.9 A simple example of a lineally convex set whose closure is not lineally convex is 
the following. Define 


A= {z € CHa 0}; Jz1| = lz2| bs A= {2 € Gia, \z1|<|z2|} 


Then A is lineally convex. Any hyperplane which avoids A must pass through the point 

(zo, 21, 22) = (1,0, 0). But this point belongs to A. This shows that A is not lineally convex. 
More generally, let I’ be a lineally convex subset of the Greek copy of Cci*\ {o} and define 

A as in Lemma 9.8.1. We can easily choose I’ without interior points but still such that the union 


UY, =CA 

vel 
has interior points. Thus! = @,CA = (CA i 2 0. Then ma A) r* cun\fol 
(see Lemma 9.8.1 and (D) in Proposition 9.8.2), but A = C((CA) . fs ea \ This shows 


that A cannot be lineally convex. 


9.8.3, Duality for functions 


A function f : C'™\{0} — R, with values in the extended real line will be called (—1) - 
homogeneous if 


f(tz) = —log |¢| + f(z), zZE chen tol, te C\{o}. 


(9.88) 
For such functions we define the Jogarithmic transform 2 f: 
(2f)(6) = sup (—log|¢-z|— f(z), ceomnsol, 
zedom(f) 
(9.89) 


We define log0 = —oo. The difference —log|¢-z| — f(z) is well-defined if f(z) < +00; 
another way to formulate the definition is to use lower addition +: 


(LAC) = sup((-tog ¢-)) +4), Ce cr\foh. 


(9.90) 
Lower and upper addition are defined in Section 9.2 on page 253. 


Proposition 9.8.10 For any homogeneous function f : cln\ {o} — R, its logarithmic 
transform & f is a homogeneous function with 


dom(2f) c (dom(f))’, 


(9.91) 


where dom(f) denotes the effective domain of f. 


Proof The homogeneity of &f is obvious from its definition (9.89). To prove (9.91), we note that 
¢ € (dom( f)) means by definition that the hyperplane Y; and the effective domain dom(f) 
have a common point z (cf. (9.87)), so that (Yf)(¢)> — log |¢- z| — f(z) = +00, thus 
¢ € dom(£f). The inclusion (9.91) may be strict as will be shown below: see Example 9.8.16 
and Remark 9.8.21. 


The analogue of Fenchel's inequality holds: 


—log|¢-2|<f2)4 (AO, Gz crr\{o}. 
(9.92) 


Moreover the usual rules for a Galois correspondence hold: f<g implies Yf>Lg, and we 
always have Y(.£f)<f. As a consequence of these two properties, 7° 2° # = £. A function 
f will be called Y -closedif 2( Lf) = f (equivalently, if it belongs to the range of Y). Some 
simple examples follow. 


Example 9.8.11 Jf f assumes the value —co, then @f is +00 identically. The same is true if f 
never takes the value +oo (n>1). If f is +00 identically, then @f is —oo identically. If 
f(z) =—log|t| when z=ta for a fixed a€C'™\{0} and +00 otherwise, then 
(LFf)(C) = —log|¢- a. If f(z) = —log|a-z| for some a, then (¥f)(¢) = — log |t|_ when 
¢ = ta and +00 otherwise. All these functions are £-closed. 


As a consequence of (9.90), we note that ee = “(inf f;) for any indexed family (f;) of 
functions. Indeed this follows from the ville up Ke alr a;)=c ke which is valid also for 


any constant c € R). This implies that any sapteriuil of 2- éigsed functions is &-closed; in 
fact, we have 


sup fj = up) = L (inf 2f)) 


(9.93) 


if the fare Z-closed. 
Homogeneous functions appear rather naturally in complex analysis. Let uw be an analytic 
functional in an open subset w of C”, u € @'(w). Its Fantappié transform is 


(Fu)(Q) = wz (Cot G21 +--+ Gaen))s 


which is a holomorphic function of ¢ € 2°, where O is the set of all z € corso} such that 
zo # O and (z1/20,.--,2n/Z0) € w. This implies that log |.Fy| is plurisubharmonic in 2”, and 


it is moreover homogeneous in the sense of (9.88). (We define it as +00 outside 2”) 

Given f defined in cl\ {o}, we can define a function F in C” by putting 
F(z’) = f(1,21,---,2n), 2’ €C”. Conversely, if F is defined in C”, we can define a 
homogeneous function fin c'*™\ {o} by 


foye F(z1/20,-+-,2n/Z0) —log|zo], z€CM"\{0O}, 2 #0; 
Z)= tees zec'\{o}, zo = 0. 


The transform (9.89) then takes the form 


(2F)(¢’)= sup (—log/l+¢'-2')—F(z’)), cre cr. 
F(z')<+00 
(9.94) 
In particular, if Fis radial (i.e., a function of ||z’||2 = 7), then the transform becomes 
(2F)(p)= sup (—log(1—pr)—F(r)), — p= ||¢"||220. 
F(r)<+00 
(9.95) 


Example 9.8.12 Take F(r)=0 when r<R and F(r) = +00 otherwise in (9.95). Then 
(LF) (p) = —log(1 — Rp), p < 1/R, and (L£F)(p) = +00, p>1/R. The second transform is 
L (LF) = F, so that F is L-closed. 


Example 9.8.13 The radial function 


F(yr)-= —Flog(1 —r?) 


is selfdual, i.e., (YF)(p) = —4+log(1 — p”). Going back to C''"\ {0}, we see that the function 


fa) = {Peel leol? = IB), ze C'"\{0}, zo] > Mela 
TOO, zeCr\{O},  |zo|<Ilz'Il2 


has this property. This function therefore plays the same role as the convex function 


f(z) = Sllell3. x € R”, for usual convexity. 


Now let A be a homogeneous set in c'*"\ {o}. We define a function d,, the distance to the 
complement of A relative to Cre tor, as 


da(z) = inf(||z — wll2; w € (C**”\{0})\ A), zeEc'*\{o}. 


(9.96) 


The function —logd, is homogeneous, and it is less than +00 precisely in the interior of A. 
Analogously we define a function d ,* by 


da-(¢) = inf(|¢— |; 8 (C1™\{0})\A°), Ce Ch*"\ {0}, 


(9.97) 


where A’ is the dual complement of A defined by (9.84). If A is empty, then d4 = 0 identically, 
whereas d 4* = +00 identically. 


Theorem 9.8.14 Let f : cl\ {o} — R) be any homogeneous function. Then 


C — log ||¢l|2<(PF)(QV<C —logda-(Q),  ¢ ¢ C*"\ {0}, 
(9.98) 


where d4* is defined by (9.97) taking A = dom/(f), and C = — inf f(z)<+ 00. We have 


||z\]2=1 
C = —oo if and only if f is +00 identically; in this case £f is —co identically. We have 
C = +00 if and only if f is unbounded from below on the unit sphere S; then &f is +00 
identically. If f is bounded from below on S, then C' < +00 and (9.98) shows that £f has at 


most logarithmic growth at the boundary of (dom(f)) ; moreover 


* 


dom(f) =(dom(f)) =(dom(¥f)) c dom(f) c (dom(f))’, 


(9.99) 
and 
dom(¥f) c (dom(f))" < (dom(f)) * 

(9.100) 
In particular dom(£f) = (dom(f)) ifdom(f) is closed. 
Lemma 9.8.15 For any subset A of C!+"\ {0} we have 

IC - z|>d4-(€)|l2Il2, ¢ect\{ol, zEA, 
(9.101) 


and 


lC-z|>MIlleda(z), Ce A", ze C\{o}. 


(9.102) 


Proof Given ¢ € C!*"\ {0} and z € A we define a = ¢ + tz where t = —||z||;?(¢- z). Then 
a:-z=0, which, if a0, means that aé CA* since ze€A._ Therefore 
dy(Q)X<||€ — allg = |¢-2|/|lzl]2, which proves the first inequality except when 
C= |lz||3?(¢-z)z. Since dy: is continuous, this restriction can be removed. If we now 
interchange the role of ¢ and z, we get |¢- z|>||¢||zd4*(z). But A” > A, so dy»(z)>da(z). 
This proves the lemma. (Interchanging z and ¢ once more, we see that (9.101) holds even for all 
zE€A) 


Proof of Theorem 9.8.14 By the Schwarz inequality and (9.101) applied to A = dom(f) we get 


— log ||¢||2< — log |¢ - z|< — logdy:(0), ¢ec\{o}, ze Ans. 


Thus 


<(— log d4-(¢)) + sup(—f); 
* ANS 


IO Bis 1982 FY 5 tog ila + sup(—f) 


The cases where + and + give different results never occur, so we can replace + by usual 
addition. This proves (9.98); note that sup(—f) = sup(—f) = —inff. 
ANS s a 2 
We already know that dom(2@/f) C (dom(f)) ; see (9.91). If ¢ € (dom(f)) and 
C' < +00, then d4-(¢) > 0 and (#f)(C)<C — log d4*(¢) < +00, so that €¢ € dom(#f). This 
proves that (dom(f))” Cc dom(£f) C (dom(f)) Taking the interior of these sets we get 


(9.99); taking the closure we get (9.100) (cf. Proposition 9.8.2). 
Example 9.8.16 The effective domain of &f may fail to be lineally convex, although it is 


squeezed in between the two lineally convex sets (dom(f)) =dom(f) and (domf)’; see 
(9.99). Indeed, let w* = (k~?,k1, 1) € C1** and define f(w*) = logk, k = 1,2,3,..., and 
f(z) = +00 when z ¢ Cw*. Then 

Ge cua\fol. 


) = +00, 


(2F)(¢) = Sup|e “Te 


Co/k + C1 + ke 


Put a = (1,0,0) and 8 = (1,1,0). Then 


(Lf)(a) = Sup(—lee ko! 


so thata € dom(£f), whereas 
(£A)(8) = sup( tog |i +1|) =0, 


showing that 8 € dom(£f). The points w* define hyperplanes 


Yur = {C5 Cok? + Gk + Co = OF, 


which converge to a hyperplane Y,, = {¢; C, = 0} with w = lim w* = (0, 0,1). By 9.99), 
(dom) =C(¥., U (UY) C dom(2f) c (dom())" = C(U¥,»). 


Both a and f belong to Y,, but as k —> +00, the hyperplanes Y,,x approach a more rapidly than 
B (note that a-w* = 1/k?, while 8- w* =1/k+1/k*). This explains why a ¢ dom(£f) 
while 8 € dom(£f). A hyperplane which avoids dom(“f) must be either one of the 
hyperplanes Y,,:, or (possibly) their limit Y,,. However, the hyperplanes Y,,x do not contain a, 
and the hyperplane Y,, intersects dom(£f) inf. Therefore there is no hyperplane which passes 
through a and avoids dom(£f). This shows that dom(.“f) is not lineally convex. In 
particular we must have (dom(f)) #4 dom(¥f) 4 (dom(f)) ; cf. (9.99). 


Example 9.8.17 A fundamental property of convexity is that the union of an increasing sequence 
of convex sets is convex. (More generally, this is true for the union of a directed family.) This is 
not so with lineal convexity. Let A, be the set of all € such that (“2 f)(¢)<k. It is easy to see that 
this is a lineally convex set; indeed, 
Ag= fh {¢ E c'*\ {0}; — log |¢- z| — f(z)<k}. 
z€dom(f) 

The union of the A, is dom(2f). If we let f be the function constructed in Example 9.8.16 we 
get an example where the A, are lineally convex but their union is not. 


Theorem 9.8.18 Let f be a function on C''"\{0} which is bounded from below on the unit 
sphere and let @f be its transform defined by (9.89). Then &f is plurisubharmonic in the 
interior of dom(£f), which is a lineally convex set. Moreover &f is locally Lipschitz 


continuous in(dom(2@f)) ; more precisely 


lim sup ZIG CDO < lols , CE (dom(Zf)) , deci, 
t0+ a 


where d 4+ is the distance to the complement of dom(£ f). 


Proof Consider the function g(¢) = — log |¢ - z]. Its gradient has length ||z||2/|¢ - z|. At the point 
a =¢+tZ, where t = —||z||;7(¢- z), g takes the value +00, so 


7 a 
daom(g(¢)< lla — Clla< 7, = Twas" 


Now £f is a supremum of functions of the form g plus a constant for various choices of z. All 
competing functions must satisfy dom(g) > dom(-/f), so that ddomg)>@4*- Therefore they 
have a gradient whose length is at most 1/d4+(¢), which implies that “ f is Lipschitz continuous 
as indicated. That &f is plurisubharmonic now follows from standard properties of such 
functions: fis a continuous supremum of plurisubharmonic functions. 


Finally (9.99) shows that (dom(2 f)) " is lineally convex: it is equal to the dual complement 
of the closure of dom(f). 


9.8.4 Examples of functions in duality 


In this subsection we shall make a detailed study of the functions 


—(1— c) log ||z||2 — clogda(z), 2 € A; 
fe(z) = 


~ ses (aae a 
(9.103) 
and 
pal) = OL ~ Hos lie — clog dar(Q), Cee | 
” Ce(Ch\{0})\4, 
(9.104) 


where 0<c<1, A is any homogeneous subset of cl"\ {o}, A’ its dual complement, and where 
d, and d ,» are defined by (9.96) and (9.97), respectively. 

We shall call fo = I, the logarithmic indicator function of the set A. Its restriction to the unit 
sphere is the indicator function in the usual sense. And “fp = “J, is analogous to the support 
function of A, thus preserving the situation from convex analysis where the support function is 
the Fenchel transform of the indicator function. We shall determine this function explicitly: it is 
y1 = —logdy:. 

More generally, it turns out that the function y1_, is essentially dual to (. It might seem 
strange to consider functions like f, which are not plurisubharmonic . We must have 


L(L fo) < fo in the interior of A. From this point of view it is more natural to consider 


(1 —c) log |zo| — clogda(z), z€ A; 


ale) = 1 ze (Cl\{0})\A, 
(9.105) 
and 
ce —(1—c) log |¢o| — clogdax(Q), CE A’; 
sae Ce (Ch"\{0})\4". 
(9.106) 


If A is contained in a coordinate patch zo 4 0 and if moreover ||z||2/|Zo| is bounded when z € A 
, then f, and g, are finite in the same set and differ there by a bounded function. If moreover 


(1,0,...,0) is an interior point of A, then ¢) 4 0 when ¢ € A’ and ||¢||2/|Co| is bounded there, 


SO @, and y, are finite in the same set and their difference is bounded there. Therefore our results 
on f, and g,.can easily be translated into inequalities for g, and y,. 
The first result is a simple inequality. 


Proposition 9.8.19 With f, and gy, defined by (9.103) and (9.104) we have £fe<yi-c for 
O0<c<l. 


Proof If ¢ ¢ A”, then y1_,(¢) = +00, so the inequality certainly holds. If on the other hand 
¢ € A’, we canestimate (f,)(¢) using Lemma 9.8.15: 


(2 fc)(¢) 


= sup [—log|¢- 2] + (1 — c) log ||z|2 + clog da(z)| 


z€A(c) 


= sup [—(1—c)log + (1 — c) log ||z||2 — clog |¢ - z| + clog da(z)] 


G*2 
z€A(c) 


< ap i — ¢)log(d4*(¢)|lzll2) + (1 — €) log ||z||2 


—clog(||¢||2d.4(z)) + clog da(z)] 
< — (1—c) logd,-(¢) — clog ||¢|l2 = y1-c(¢). 


The supremum is over the set A(c) of all z such that f.(z) < +00, that is 


A f = 0: 
Ae { _ ee, 
A for0<c<l. 


(9.107) 
(A(c) can be empty; in that case “f, is —oo identically.) 


We now study inequalities in the other direction. The cases c = 0 and c = 1 are easy and will be 
considered first. 


Proposition 9.8.20 For any homogeneous subset A_ of cto} we have 
LI, = L fo = v1 = —logd,:. (The analogue of the support function of A.) 


Remark 9.8.21 Note that here dom(@fy) = A’ = A” (dom(f),) is open and lineally 


convex, whereas (dom(f),) = A’; again we see that the inclusion dom(£ f) C (dom(f)) 
may be strict (cf. (D) in Proposition 9.8.2). 


Lemma 9.8.22 Assume that A is homogeneous and not empty. For every ¢ € A’ there is a 
point z € OA, z #0, such that |¢ - z/<d 4*(¢)||z|l2. 


Proof For every ¢ € A” there is a point a € 0A, a £0, such that ||a — ¢||2 = d4+(¢). Thus 
a¢ A’ =A’ (cf. (D) in Proposition 9.8.2). Now a ¢ A” means that Y, 7A 0 (see (9.85) 


and (9.87)). On the other hand a € 0A’ C Lea’ (cf. (C) in Proposition 9.8.2), so that 
Y,A = 0. Therefore Y,, meets the boundary of A, and we can choose z € SOA such that 


a+ z=0.Then |¢- 2| = |(¢ — a) - z|<||C — alla - |l2ll2 = da ()|l2ll2- 
Proof of Proposition 9.8.20 If A = 0, we have YI, = —logd 4+ = —oo. Otherwise the lemma 


provides us, given any ¢ € A’, with a point z € 0ANS such that 


(£ fo)(¢) = sup(— tog | , W 41g l|uilla) = 1og |‘ ve ios als 
> — logd,+(¢) = v1(¢). 


For ¢ € (C##"\{0})\A™ both fy and g, take the values +00. Thus Yfy>y, everywhere. 
The inequality & fy<y, was proved already in Proposition 9.8.19. 


Proposition 9.8.23 Assume A that is open and not empty. Then there is a constant M, which 
depends on the geometry of A, such that 


yo =Iyne Lf, = £(-logd,)>I,- — M. 


In fact M can be taken as inf j.= inf (— log d,), where as before S is the unit sphere. 


Here dom(“f,) = A’ = (dom(f),) is closed and lineally convex; cf. (9.99). 


Lemma 9.8.24 Assume A has a nonempty interior and take any point z € A . Then there is a 
constant C such that |¢ - z|<C\|¢||2d.4(z) for all C. 
Proof Given z € A define C' = ||z||2/d4(z). We have 


IC- 2l<|I¢llallzll2 = Cll¢llada(z). 


The best choice is a point z € S such that d4(z) = supd,, so that C = 1/supd,y. 
Ss S 


Proof of Proposition 9.8.23 Using the lemma above we get for any ¢ € A’, 


(2f1)(C) = sup(—log 


wea 


> — log C — log ||¢||2 = vo(¢) — M. 


¢-z] + log da(z) 


cu] + log da(w))> — log 


When ¢ ¢ A’, there is a point z € A such that ¢- z = 0, and since A is open, f(z) < +00, so 
that (£f,)(¢) = +00. Thus we have Yf;>o — M everywhere. The inequality I 4»->Lf; was 
already proved in Proposition 9.8.19. 


Theorem 9.8.25 Let A be an open homogeneous set. Then A is lineally convex if and only if 
—logdy, is £-closed. 


Proof If A = B’, then Iz = —logd,« = —logd, by Proposition 9.8.20, so —logd, is Y- 
closed. Conversely, Proposition 9.8.23 shows that #(—logd,)>I,4» — _M, which implies 
L(—logda)< —logd,» + M. Therefore, if z belongs to the open set A™ (cf. Proposition 
9.8.2), then #(—logda(z)) is finite. If —logd, is #-closed, this is equivalent to — log d4(z) 
being finite, which implies z € A. Thus A™ C A; this inclusion means that A is lineally convex. 


Theorem 9.8.26 A closed lineally convex set A can be recovered from £I,4. Indeed, if A is a set 
with these properties different from cl\ {fo}, then I~4>LI,>I4 —M, so that A is the set 
where £I 4, is finite. If A is equal to c'™\ {o}, then I, is —oo identically. If A is a closed 
and lineally convex set such that ||z' ||2<R\zo| for all z € A, then ZI4>I, — log V1 + R?. 


This theorem is thus analogous to the result in convexity theory which states that a closed convex 
set can be recovered from its support function. By way of contrast, an open set A can be 
recovered from J, only under special conditions, since @I4 = “Ij. If A is open and equal 
to the interior of its closure, and if its closure is lineally convex, then A is the interior of the set 
where #J, is finite. But an open lineally convex set can always be recovered from “(— log dy) 
; see Proposition 9.8.23. 


Proof If A is closed, lineally convex and not equal to all of chm tol. then A* is open and 
nonempty, so we can apply Proposition 9.8.23 to A’ and obtain 


Ty 2L(- log d4+)>I 4= —M. 
From Proposition 9.8.20 we have “I, = — log d,+. Combining this information we deduce that 
I, >L(—logd,*) = LI 4ST, = M. 


Since A is lineally convex, A = A™, and we see that ZI, and I ', are finite in the same set (and 
differ there at most by a bounded function). 

The last statement follows if we keep track of the constant in Lemma 9.8.24. Alternatively we 
can compare I, with the function F(z) = —log|zo| when ||z’||2<R|zo|, defining F(z) = +00 
otherwise. This gives F(z)>I,4(z)>F + log|zo|—log||z||, for z¢A, so that 
I,>F — log V1+ R? everywhere, implying that ZI4>YF — log /1+ R?. Since YF = F 
(cf. Example 9.8.12), we can conclude that I4>F — log V1 + R?>I4 — log V1+ R? in A. 


Finally we shall deduce an estimate from below for “&f, when 0 <c< 1. We shall need a 
definition. 


Definition 9.8.27 We shall say that A satisfies the homogeneous interior cone condition if there 
exist positive numbers y and R such that for every b € OA and every r< R, the inequality 


sup(da(z); ||2 — blla<r|[bll2)>r lalla 


holds for every b € OA and every r<R. 


Proposition 9.8.28 Assume that A is open, nonempty, and satisfies the homogeneous interior 
cone condition just defined. Then there is a constant M such that ~\--2Lfe->pi-c — M for 
every c € [0,1]. 


Here dom(£f,) = (dom(f.)) for 0<c < 1, whereas it is closed for c = 1 as already noted. 

In particular a set with Lipschitz boundary satisfies the homogeneous interior cone condition. 
To prove this proposition we shall need a lemma which combines Lemmas 9.8.22 and 9.8.24. 
The requirements concerning the point z are somewhat contradictory, since z € OA in the first 
and z € A in the second. Nevertheless, we can find a compromise: 


Lemma 9.8.29 With A as in Proposition 9.8.28, there exists a constant C such that for every 
¢ € A’ there is a point z = z¢ € A such that 


IC - 2|<Cl|Cllada(z) and |¢ - 2|<Cd4-(¢)|l2\l2. 


Proof First pick any point w € A. It will serve as the point z; for all ¢ such that d4+(¢)>R\|¢|l2: 


l¢-w]<Ii¢llalleoll2 = 22 |\¢llada(w)<Clg|lada(w) 


and 


¢-w]<cllallells = Fd a-(6)llulla< ear (0)|lell2<Caa-(¢) hulle 


for a constant C> max(R™, ||w||2/da(w)). 


The case d4-(¢)<R\|¢||z remains to be considered. To a given ¢ € A® we choose a € OA’, 
a #0, such that |la — ¢|/2 = d4*(¢) =r||¢ll2, r<R. Since Y, meets OA (cf. the proof of 


Lemma 9.8.22), we can choose a € OA, a # 0, such that a- a = 0. The homogeneous interior 
cone condition now implies the existence of a point z = z¢ € A such that d4(z)>yr|la||2 and 
2 — all2<rllal|2<Rlla|l2. Then |¢-a| = |(¢—a@)-al<||¢ — allallall2 =da-(¢)llall2 and 
|| — all2<rllall2 = |lal|2d4-(¢)/II¢ll2, so that 


6-2] = |¢-a + ¢(z—a)|<|¢- a] + [[ell2|z — a] <2d4-(Q)|lalla< zag a ()llalle. 


Here the last inequality follows from ||z — a||x2<Rlla||2; it is no restriction to assume that R < 1. 
On the other hand d4(z)>yr|la||2 = y\|a||2d4*(¢)/||¢||2 so that 


Ic 2|<2d4-(C)|lall2<2da(2)ll¢llo- 


With the constant 


= 1 whe 2 2 
C= max| F, d,(w)’? 1-R? ¥ 


this proves the lemma. 


Proof of Proposition 9.8.28 Since A is open, dom(f,) = A for all c; cf. (9.107). Using the 
lemma we get for any ¢ € A’, taking z € A from Lemma 9.8.29, 


(Lf-)(¢) =sup(—log 


weA 
> —log|¢- z2| + (1 — c) log ||z||2 + clog d(z) 
= (1 — e)(log ||z||2 — log |¢ - z|) + c(log da(z) — log |¢ - z|) 
>(1 — c)(log ||z||2 — log(Cd 4-(¢)||zIl2) 
+ce(log d4(z) — log(C|||¢||2d.4(z)) 
= —(1 —c) logd4:(¢) — clog ||¢||2 — log C = yi_--(¢) — M. 


If ¢ ¢ A’, then (Yf.)(C) = y1_-2(¢) = +00. 


9.8.5 The support function of a convex set 


C-u] + (Lo) log lla + log da() 


Let Ag be a subset of C”. Its support function is 


Ha,(¢') = sup Re (¢’: 2’), 4 = (cinerea) EC". 


Z'CAo 
(9.108) 


If Ap is closed and convex, then Hy, determines Ag; in fact, Ag is the set of all z’ = (z1,..., Zn) 


such that Re(¢’-2’)<Hy4,(¢') for all ¢. Now let A be the homogeneous set of all 
ZE Cc ™\ so} such that zo 4 0 and z’/zo € Ao. With the set A we associate the function “I, 
defined by 


ZzeA 


(-2I4)(¢) = — log da*(¢) = sup(— log | -2)+log|lzll2),  ¢«€ crm\fol 


(9.109) 


the projective analogue of the support function. What is the relation between these two support 
functions? To answer this question we first modify J, a little and define 


); ‘a= crn\fol 


If Ag is bounded, then hy and “J, are finite in the same set and differ there by a bounded 


function. 
We shall express f, in terms of H4,. We first formulate an auxiliary result, which we shall 


+ log |Z 


ha(@) = sup(—loe|¢-# 


(9.110) 


need for convex sets in the complex plane only, but which is also valid in R”. We shall therefore 
use the real support function 


H,(é)=supé-2,  €€R”. 


reA 
(9.111) 
Lemma 9.8.30 Let A be a convex set in R”. Then 
—inf|/z|lo< inf H <inf ||x 
nf ||el|a<, inf Ha(é)<inf \a\| 
(9.112) 


with equality on the left if0 ¢ A , and on the right if0 € A. 


Proof For any set A we have, writing S for the unit sphere, 


infH 4 = infsup€é - x>supinff£ - x = sup(—||z = —inf||zrllo. 
nf Ha = infsupg - x>supinfé © = sup(—le||2) = —inf || 


If A is convex and x ¢ A, then there is a € € S such that €- e>H4(&); thus ||x||2>€-«>Ha4(E), 
so that ||z|| 22inf H A. This shows that a l|z|| 22inf H 4 and proves (9.112) for all convex sets. 
x 


Now assume that 0 ¢ A. Then A must be contained in a half-space {x; £- x>c} for some 
€€S and c>0, which shows that H4(—&)< —c<0. If A is empty, (9.112) has the form 
—oox< — oo<0, so the result is true. If A is not empty, then we can choose c = inf||2||2, so that 

LE 


inf H. AX —-c=- inf ||z||2; we have proved equality on the left in (9.112). 
LE 


On the other hand, if 0 € A and H A(€) < c for some € € S and some c, then necessarily 
c > 0 and the vector c€ cannot belong to A, so that inf||2ll2<lle&ll2 = c. Thus inf||2||2<inf Ha 
x x 


; we have proved equality on the right in (9.112). 
Lemma 9.8.31 For any convex set A in R” we have 


inf ||z||,= sup Ha(€) ; 


ace léll2=1 

(9.113) 
inf ||z||2 = inf oe 
inf ll2 nes A(&) 

(9.114) 


where t* = max(t, 0), = max(-—t,0) fort € R. 


Proof If 0 ¢ A, then (9.113) is just the first part of (9.112) with equality, and (9.114) reduces to 
0 = 0. If 0 € A, then (9.113) reduces to 0 = 0 while (9.114) is the second part of (9.112) with 
equality. 


Proposition 9.8.32 Let Ag be any convex set in C", and A the homogeneous set of all 
z€C'"\{0} such that z) #0 and 2'/z9 € Ao. Define Ha, and hy by (9.108) and (9.110), 


respectively. Then 


ha(¢) = inf log (Ha,(t¢') + Re (t¢o)) _, ¢ € crn\fol 


\t|=1 
(9.115) 


Proof If A is empty, (9.115) certainly holds, because both sides are equal to —oo. Fix 
¢ € C1*"\ {0} and denote by L the linear mapping 2’ > ¢' - z’. If A is not empty, then 


e hal) = inf 
Z'€Ag 


hi ot| os tee 
CoC =2 jm sore 


Co + a(Co) 


’ 


(9.116) 


where a(€o) denotes the point in the closure of L(A) which is closest to —¢o. Here the first 
equality holds because, in view of the homogeneity, it is enough to let z vary with zo = 1 in the 
definition of hy. 


We now note that 


H,,(t¢') = sup Re (t¢’- z') = sup RetL(z’) = sup Rets = Hy a,)(t), 
z'€Ag z'€Ag s€L(Ao) 


which shows that the support function of the set M = L(Ao) + Co is 
Hy(t) = Ha,(t¢’) + Re (to), tec. 


We can apply (9.113) to the convex set M. This yields 


eM) = 19 + a(Co)] = sup Halt) = sup(H4,(t¢') + Re (to)) 


|t|=1 


where the first equality is that of (9.116), and thus proves (9.115). 


Conversely, we can express H 4, in terms of hy. 


Proposition 9.8.33 Let Ag be a bounded but not necessarily convex set in C", and A the set of 
all z € C'*"\ {0} such that z) # 0 and z'/z) € Ao. Define Ha, and hy by (9.108) and (9.110), 
respectively. Then 

Ha,(¢') = lim (—G) — e749), Gh ©”. 


¢oER 
¢9 7-00 


(9.117) 


Proof We can still use (9.116) even though Ag now is perhaps not convex, if we let a(¢o) denote 
one of the closest points to —Co in the closure of L(A). Let Cp be real and tend to —oo. Then 


eal) + Cy = |e + (C0) | + Co + —Rea(—oo), 


where a(—oo) is an accumulation point of a(¢o) as Co € R and ¢y) — —oo. It is a point in the 
closure of L( Ao) which satisfies 


Ha,(¢') = sup Re(¢’- 2’) = sup Res = Rea(—oo). 
2'€A0 s€L(Ao) 


This implies (9.117). 


9.8.6 The dual functions expressed as a dual complement 


In convexity theory, the Fenchel transform generalizes the support function: the support function 
(9.111) is just the Fenchel transform of the indicator function. Conversely, we can express the 


Fenchel transform f of a function fin terms of the support function if we add one dimension: by 


definition we have f(£) = sup(€-a2— f(zx)), and we see that f(€) = Hepis(€,—1), where 
Hepif 18 the support function of the finite epigraph of f i.e., 


Hepis(E.n) = sup (€-a+m; f(z)<y), (6) <¢R"™xR. 
(z,y)ER"xR 


We already know that the dual complement A” of a closed set A can be expressed in terms of 
the dual function (indeed, A” is the set where YJ, is finite; see Proposition 9.8.20). Conversely, 
we shall see here that we can express the dual function in terms of a dual complement if we go 
up one step in dimension (cf. (9.130) below). The functions will then give rise to Hartogs sets, 
which we proceed to discuss. 

Hartogs domains and complete Hartogs domains were defined in Section 9.4 above. We shall 
generalize this in two ways: first we shall need to study sets that are not necessarily open; second, 
it is natural to add a hyperplane at infinity and look at subsets of projective space. Thus we 
consider sets A C (Cl*\ {0}) x C that are homogeneous in the sense of SubSection 9.8.3, i.e., 
such that (sz, st) € A if (z,t) € A and s € C\{0}. We shall say that A is a complete Hartogs 
set if (z, t’) belongs to A as soon as (z,t) € A and |t'|<|t|. Such a set is therefore defined by an 
inequality |t| < R(z) or |t|<.R(z) for some function R with O< R< + oo. We shall however use 
f = — log R to indicate the radius of the disks. 


Definition 9.8.34 Let f : c'*\ {o} — R, be a homogeneous function and X a homogeneous 
subset of cln\ {fo}. We associate to f and X a homogeneous complete Hartogs set E(X; f) in 
Clin: it is the set of all (z,t) € (C1*"\{0}) x © such that |t|<e4 when z € X, and 
[| <e fl whenz ¢ X. 


The fiber of E(X; f) over zis thus the whole tplane if f(z) = —oo; it is a closed disk of finite 
positive radius if z € (dom(f)) M X and f(z) > —oo; it is an open disk of finite positive radius 
if z € (dom(f))\X and f(z) > —oo; it is the origin if z € X\dom/(f); finally, the fiber is 


empty if z¢ XUdom/(f). If X1 Cc X2 and fi>f2, we have an obvious inclusion 
E(X1; fi) C E(X9; fe). 

Every complete Hartogs set A is of the form E(X; f) for some X and some f we can take X 
as the set of all z such that the fiber is not open and define f(z) as the infimum of all real 
numbers c such that (z,e°) belongs to A. A complete Hartogs set defines the set 
XM {z; f(z) > —oo} uniquely: if f(z) > —oo, then z € X if and only if the fiber over z is 
closed and nonempty. On the other hand the choice of XM {z; f(z) = —co} is immaterial in the 
definition of E(X; f).® 


81 get uniqueness, one could for example require that X always contain {z; f(z) = —oco} or that these two sets be disjoint, or 
else take the Riemann sphere as the fiber over points in X such that f(z) = —oo, but we shall refrain from doing so. 


Theorem 9.8.35 Consider the dual complement of E(X; f), 


E(X;f)’ = {(<, re cies to}, C-z+7t ¢ Oforall (z,t) € E(X: fy}. 
(9.118) 


If both X and dom(f) are empty, then also E(X;f) is empty and E(X; f) is equal to the 
whole set C1+"*1\ {0}. If on the other hand X U domf # 0, then E(X; f) is nonempty and its 
dual complement E(X; f) is a subset of (C!*"\ {0}) x C and a complete Hartogs set, thus 


E(X; f) =E(55¢) 
for some set & and some function 9. Here the function 9 is uniquely determined: 


(Lf)(C) when¢ € (X Udom(f))’, and 


eld) = +00 when¢ € (ch\foly\(xu domf) ; 


(9.119) 
thus p = £@f as soon as dom(Lf) C (X U dom(f)) , in particular if X C dom(f). 


We define the set © as follows. We let ¢ € & if and only if¢ € (XU dom(f)) and either f 
takes the value —oo or 


inf C-+z ef@> inf C-w ef (wv) 
z€dom(f)nX wedom(f)\X 
(9.120) 
or else 
U2exXnd h, . Pleflz") inf led). 
forall z’ € om(f) wehave |¢ - z°\e = ei ¢-zle 


(9.121) 


If f is not +00 identically, then & is uniquely determined, so that this is the only set which 
satisfies E(X; f) = E(&;). Moreover, we always have 


B(X; f)’ 1 (e%™\{0}) x {0}) = (KUdom(f))’ x {o}, 
(9.122) 


which proves that = Udom(y) = (X U dom( fy)’. The particular cases when X is empty or 
equal to dom(f) are of interest. If domf # 0 we have 


E(0; f) = E((dom(f)) ; 2) > E(dom(£f);-f). 
(9.123) 


If dom(f) is closed and nonempty and f is lower semicontinuous and never takes the value —oo, 
then 


E(dom(f); f) = E(0; 27). 


(9.124) 
Remark 9.8.36 If X Udom(f) # 0, then E(X; f) contains 
(E(; 2) ((X\dom(f))" x (C\{0}))) U(X Udom(f))" x {0}) 
and is contained in 
E((dom(f)) U(X Udom(f)) ; 2/). 
If X is a subset of dom(f), then p = Lf and these inclusions simplify to: 
B(0;-2f)U ((dom(f))" x {0}) c H(% f)" c E((domf)"; 7). 
(9.125) 


We also note the following two special cases. If f = +00 identically, then 
E(X;+00) = X x {o} and E(X;-+00)" =X". 


In this case the definition of = in the theorem yields 5 = X”. 
We have 


E(X; f)’ =(XUdom(f))" x {0} = E((X Udom(f))'; 2f), 


(9.126) 


if f assumes the value —oo. 


Proof of Theorem 9.8.35 If X Udom/(f) is empty, then E(X;f) is empty, and its dual 
complement is the whole space except the origin. If X Udom/(f) is not empty, then the 


hyperplane (C1*"\{0}) x {0} cuts E(X; f), so that no point (0,7) belongs to E(X; f), 
which therefore is contained in (C'™"\{0}) x C. 


We need to find the conditions for (¢, 7) to belong to F(X; f). This happens precisely when 
¢-z+7t is non-zero for all (z,t) € E(X;f). The case t=O is easy: we find that 


(¢,0) € E(X; Z) if and only if ¢-z40 for all z€ X Udom/(f), thus if and only if 


CE(XU dom(f)) . This proves (9.122). Now let r 4 0. Then we see that (¢,7) € E(X; f) 
precisely when the following three conditions hold: 


I7| < I< : zlef) for all z € (dom(f)) NX; 


(9.127) 
I7|<|¢- wle/™ for all w € (domf)\X; 
(9.128) 
|¢- z| # 0 for all z ¢ X\dom(f). 
(9.129) 


Fix ¢ € (X U dom( f)). We see that the three formulas (9.127)-(9.129) imply that 


I7l< exp(—(2F)(Q), 


and that they are implied by |7| < exp(—(&f)(C)). This shows that g is as described in (9.119), 
and it only remains to be seen when the inequality |7|< exp(—(Y&f)(C¢)) is strict. The condition 
on t means that it shall belong to all open disks of radius ic : z\ef (2) for z € (dom(f)) MX, and 
all closed disks of radius I< : wie! (”) for w € dom(f)\X. Now an intersection of a family of 
closed disks is always closed, and an intersection of nonempty concentric open disks with finite 
radii is closed exactly when it contains, along with any disk, also a disk of strictly smaller radius. 
This is what is expressed by conditions (9.120) and (9.121). Finally (9.123) and (9.124) follow 
from an analysis of (9.120)-(9.121) in the special cases X = 0 and X = dom(f). 


Corollary 9.8.37 The logarithmic transform £ f of any function f can be obtained from the dual 
complement of E(Q; f): it is minus the logarithm of a certain distance, viz. the distance from 


(¢, 0) to the complement of E(0; f) in the direction (0,...,0,1): 


(LF)(C) = —logtint(|r|; (¢,7) € CH" \E(0; #9"). 
(9.130) 


Proof This follows from (9.123). The result also explains why we cannot expect these functions 
to be pullbacks of functions on projective space. 


9.8.7 Lineally convex Hartogs sets 


Intuitively, it seems that E(0; f) and E(dom(f); f) ought to be lineally convex simultaneously. 
This is not quite true. We shall note three results in the positive direction, Propositions 9.8.38— 
9.8.40 below, and one result in the negative direction, Example 9.8.41. Then we shall establish 
conditions under which it is true that fis Y-closed if and only if E(dom(f); f) is lineally 
convex (Corollary 9.8.43), as well as conditions which guarantee that fis -closed if and only if 
E(0; f) is lineally convex (Theorem 9.8.49). 


Proposition 9.8.38 If E(X;f) is lineally convex, then also XUdom(f) and 
E(X Udom(f); f) are lineally convex. In particular, if E(; f) is lineally convex, then so are 
dom(f) and E(dom(f); f). 


Proof Suppose that E(X;f) is lineally convex. That X Udom/(f) is lineally convex then 
follows from the easily proved result that the intersection of a lineally convex set and a complex 
subspace is lineally convex as a subset of the latter. If E(X;f) is lineally convex, then also 
E(X; f +a) is lineally convex for any real number a. Any intersection of lineally convex sets 
has the same property, so we only need to note that E(X Udom/(f); f) is equal the intersection 
of all E(X; f —a),a> 0. 


Proposition 9.8.39 If f is upper semicontinuous and there exists a set X such that E(X; f) is 
lineally convex, then E(Q; f) is lineally convex. 


Proof We know from Corollary 9.8.4 that E(X;f) is lineally convex if E(X; f) is lineally 


convex. Now E(X; f) = E(Q; f) if fis upper semicontinuous, hence the result. 


However, the semicontinuity of fis not important—it is the fact that the effective domain is open 
which is relevant. This is shown by the following result. 


Proposition 9.8.40 If XUdom/(f) is open and E(X;f) is lineally convex, then 
E(X\dom(f); f) is lineally convex. In particular, E(Q; f) is lineally convex ifdom(f) is open 
and F(X; f) is lineally convex for some subset X of domf. 


Proof Assume that E(X; f) is lineally convex. Then also E(X; f + a) is lineally convex for any 
real a, and we shall prove that the union of all E(X;f+1/k), k=1,2,..., which equals 
E(X\dom(f); f), is lineally convex. 
The hyperplanes in C't”*? will be denoted by Y(¢,7) in analogy with (9.85), thus 
Yor = {(z,t) € (C*™ x C)\ {0}; ¢-z+7¢ =O}. 


(9.131) 


Let (z,t) € E(X\dom/(f); f) be given with z € X Udom(f). For any k there is a hyperplane 
Y(ck,-*) that contains the point (z,t) and which does not meet the set F(X; f + 1/k). We may 
assume that ||C*||? + Faalle = 1. Take an accumulation point (¢,7) of the sequence (¢*,7*). 
Since X Udom/(f) is open, we can be sure that tr 4 0. The hyperplane Y(¢7) passes through 
(z,t) and does not meet E(X\dom/(f); f) since 7 ¥ 0. If on the other hand z ¢ X Udom(f), 
there is a hyperplane Y(¢,9) which passes through (z, t) and does not cut E(X; f). 


The openness in Proposition 9.8.40 cannot be dispensed with as we shall see now. 


Example 9.8.41 There is a function f such that E(dom(f); f) is lineally convex while E(Q; f) 
is not. Define 


R(z)= inf |(k+1)z,— 2 —2o/Al, = (z,2;) € CM\ 04, 
(2) =, int |(-+ 1a —z0-zo/kl, — 2= Gnas) Cfo} 


and let f =—logR. Then dom(f) consists of the complement of the hyperplanes Y;, 
¢ = (1, —k), and 


E(dom(f); f) = A(z, t); z€ Yaj-nand | <| +1)z1 —z9- za/¥\} 


is lineally convex. (Note, however, that E(X; f) is not lineally convex if X contains dom(f) 
strictly; cf. Example 9.8.9.) The function f is £-closed; cf. (9.93) and Theorem 9.8.42 below. To 
prove that E(Q; f) is not lineally convex, let us note that (1,0,1) ¢ E(Q; f), for R(1,0) = 1. 
Suppose there exists a hyperplane Y(¢,,) which passes through the point (1,0, 1) but does not cut 
E(0; f). Then Gy + 7 = 0. We must also have t # 0 since (1,0,0) € E(0; f) as well as ¢, £0 
since (1, 2,1) € E(Q; f). Moreover 


inf 


(k+1)z1 — 20 - zo/k| forall z. 


Taking z = (¢1,—¢o) we see that there is a number m such that ¢; = —mo, and we can 
conclude that, taking z) = 1, 


\¢-2| 
ol 


=m 


A 1/m|>int|(k +i) 1— 1/h|. 


However, for z, close to 1/m we must have 


’ 


inf (k +1) -1- 1/k = (m+ 1)z; -1-—1/m 


so that 


m|z1 — 1/m|>(m + 1)|z1 — 1/m| 


for all z, close to1/m. This is impossible, which shows that there is no such hyperplane. 


Theorem 9.8.42 If f= Lf in (XUdom(f))", then E((XUdom(f))”; f) is lineally 
convex. In particular, if we assume domf to be lineally convex and X C dom(f), then 
LL f = f indom(f) implies that E(dom(f); f) is lineally convex. 

Conversely, if E(X; f) is lineally convex, then f = 2(Lf) in X Udom(f), which is a 
lineally convex set. If f is bounded from below on the unit sphere, then f = ¢(Lf) = +00 


outsidedom(f) . Thus in this case (Lf) = f everywhere ifX > dom(f) \dom(f). 
Proof Suppose that f = Lf in (X U dom(f)) Take 


(2°, 2°) ¢ E((X Udom(f))”; f). 


We shall then prove that there is a hyperplane Y(¢,,) (see (9.131)) which contains (2°, t°) and 


*K 


does not cut E((X Udomf)  ; f). Consider first the case 
z? €(XUdom/(f)) 


We know that je°| > ef (2), By the definition of &f and __ since 
(LLF)(2°) = f(z) > —log |e° 


—log|¢- z°| — (LFf)(¢) > — log |t°]. 


Then we take r = —C- z°/t°, so that (z°,t°) € Yic,) and —(-“f)(¢) > log |r|. Moreover, for 
any (z,t) € Y(¢,7) we have 


f(a) 2(PLf)(z)> — log |¢- z| — (LF)(¢) = — log |rt| — (-LF)(¢) > — log lt, 


which shows that (z,t) ¢ E((XUdom(f))” ; f). The case 2° ¢ (X Udom/(f)) remains to 
be considered. In this case there is a hyperplane Y; that contains 2 and does not meet 


(X Udom(f))”, so the hyperplane Y(¢,0) does not cut B((X U dom(f))”; f). 

Now assume that E(X; f) is lineally convex. We already know that X U dom(f) is lineally 
convex (cf. Proposition 9.8.38). If © f assumes’ the value —oo, _ then 
E(X; f) = (X Udom(f)) x C, and f = Lf = —co in X Udom(f). If f > —oo, let 2° be 
any point in X Udom/(f) and take ? such that |e°| > ef), thus (z°,t°) ¢ E(X; f). By 
hypothesis there is a hyperplane Y(¢,,) which passes through (z°, t°) and does not meet E(X; f). 
Since (z°,0) € E(X,f), we must have t #0, so we obtain a minorant of f of the form 
— log |¢- z| + log|r|<f(z), where the left-hand side takes the value —log |t°| < f(z®) at the 
point 2 and moreover can be chosen larger than any number less than f(z°). Thus 
(LL f)(z°)>f(z°) and we conclude that YY f = f in all of X Udom(f). 

Finally, assume that f>—C' on the unit sphere without any further assumption. Thus, 
putting A = dom(f), we have f>g = I4 — C, so that YPf<Lg = C — log dy: by Proposition 
9.8.20. We now note that dy» = d,y« and take the transformation once again, this time using 
Proposition 9.8.23. We get PL f>rLlLg>l,*» —M—C. In particular & f(z) = +00 if 


z¢A =A” =dom(f) (ef. (D) in Proposition 9.8.2). This finishes the proof. 


, we can choose € such that 


* 


Corollary 9.8.43 Assume that f is bounded from below on the unit sphere and that dom(f) is 
closed and lineally convex. Then f is £-closed if and only if E(dom(f); f) is lineally convex. 


We now proceed to study the case when dom(f) is open and f tends to +00 at the boundary. 
Propositions 9.8.44 and 9.8.47 below are applicable when f tends rather fast to +oo, and 
Theorem 9.8.49 in a more general situation. 


Proposition 9.8.44 Let f be a homogeneous function on C1*"\{0} which tends to +00 at the 
boundary of dom(f) = A in the strong sense that f> — C' — log d, for some constant C, where 
d, is defined by (9.96). Then f is -closed if and only if E(0; f) is lineally convex. 


Proof If fis -closed, then its effective domain A= A must be lineally convex by Theorem 


* 


9.8.14, for A = (dom(7Lf)) =dom(“f) in view of (9.99) applied to Yf (this function 
is bounded from below on the unit sphere unless fis +00 identically, a trivial case). Theorem 
9.8.42 now shows that F(A; f) is lineally convex and Proposition 9.8.40 implies that E'(Q; f) is 
lineally convex. 

Conversely, assume that E(Q; f) is lineally convex. In view of Theorem 9.8.42 it only 
remains to be proved that YL f = f = +00 outside A = dom(f). Now if f>—C — logdya, 
then YF<C + L(—logd,)<C + I,» by Proposition 9.8.23. We take the transformation again 
and obtain YL f> —C+ LIy4* = —C — logdy*, using Proposition 9.8.20. But dom(f) is 
lineally convex, so. A” = A. Hence Y Lf = +00 in the complement of A. 


Functions with bounded logarithmic transforms exhibit the behavior studied in Proposition 
9.8.44: 


Proposition 9.8.45 Let f be a homogeneous function on C1*"\{0} such that dom(f) = A 
equals the interior of its closure and such that dom(f) is lineally convex. Assume that f is 


bounded from below on the unit sphere and that £ f is bounded from above in SM dom(# f). 
Then f>— C' — log d,4, where C is a constant. 


Proof We have LF<C + Ip, where B=dom(f). Therefore 
feL(Lf)>-—C+ LIp = —C — logdg: = —C —logdg:. The next lemma shows that 


* 


B =A. 
Lemma 9.8.46 Let f be a homogeneous function on C'*"\ fo} such that 
dom(f) = dom(f). 


Assume that f is bounded from below on the unit sphere and that dom(f) is lineally convex. 
Then (dom(£f)) = dom(f). 
Proof From (9.99) we deduce, recalling that fis bounded from below on S, that 


* 


dom(f) "5 (dom(“f))” > (dom(f)) 


kK 


Now, since domf is lineally convex, so is its interior dom(f) = dom(f) (Corollary 9.8.4). 
Therefore dom(f) > (dom(¥f)) > dom(f). Taking the interior of these sets we get 
(dom(f))" = dom(f). 


Under a regularity assumption we can let ftend to infinity at a slower pace: 


Proposition 9.8.47 Let f be a homogeneous function on C1*"\{0} which tends to +00 at the 
boundary of dom(f) = A in the sense that f>—C —clogd, on the unit sphere for some 


constants C and c with 0 < c<1. Assume that A” satisfies the homogeneous interior cone 
condition (Definition 9.8.27). Then f is 2-closed if and only if E(Q; f) is lineally convex. 


Remark 9.8.48 It can be easily proved that if A is lineally convex and its dual complement A’ 
satisfies the homogeneous interior cone condition, then so does its set-theoretic complement CA. 


Proof In view of Theorem 9.8.42 and the proof of Proposition 9.8.44, it only remains to be 
proved that YL f= f =+00 outside dom(f) if E(@;f) is lineally convex. Now if 
f= —C —clogdy on the unit sphere S, then we obtain f> — C+ f, everywhere, introducing 
the function f, of (9.103). We take the logarithmic transformation once to obtain 
LF<C+Lf-<C+y1-- (Proposition 9.8.19), and then again to _ get 
LLfS—-C+Lpi--> —-C-—M-+ f,, this time applying Proposition 9.8.28 to the function 
(~i1—c and using the homogeneous interior cone condition (Definition 9.8.27) on A’. This shows 
that YL f equals +00 in the complement of A. 


We finally come to the general case of a function which tends to infinity at the boundary. 


Theorem 9.8.49 Let f be a homogeneous function on C1+"\ {0}. Assume that f is bounded from 
below on the unit sphere and tends to +00 at the boundary of A = dom(f) in the sense that 
A, = {z€ S; f(z) < s} is strongly contained in dom(f) for all numbers s, i.e., the closure of 
A, is contained in the interior of A. (This implies that A 1s open.) Assume moreover that A 


satisfies the homogeneous interior cone condition. Then E(0; f) is lineally convex if and only if f 
is £-closed. 


Proof For the proof we shall need the functions f4,, where A is a homogeneous set and ris a 
positive number, defined as fa, = —logr+ I,, thus fa, = —logr — log||z||, when z€ A 
and f4, = +oo otherwise. We note that “f 4, = logr —logdy: (Proposition 9.8.20, so that 
(¢, 7) belongs to F(A"; #f4,) if and only if ¢ € A® and r|t|<d4+(6). 

What remains to be done, considering Theorem 9.8.42 and the proof of Proposition 9.8.44, is 
the following, assuming E(Q;f) to be lineally convex. Given any z° ¢ dom(f) = A it is 
required to find a hyperplane Y(¢,-) with 7 #4 0 which does not cut (0; f) and passes through 
2", t°) with \t°| arbitrarily small; the problem is to avoid the vertical hyperplanes, those with 
T = 0. Since fis bounded from below on S, there is a number R such that (0; f) is contained in 
E(0; fa.r). On the other hand, given any € > 0, there is a homogeneous set K which is strongly 
contained in A and such that f>-— loge on S\K; this means that E(@; f) is contained in 
E(0; fa) U E(Q; fx,r). We shall find a hyperplane which does not cut the latter set for a 


suitable choice of ¢. This amounts to finding (¢, 7) in E(0; f he) N E(O; fr, r) » equivalently in 
E(A*; ffae) NE(K’; fx.p); cf. (9.123). The hyperplane shall also contain a point (2°, t°) 
with |t°| = 6 positive but arbitrarily small. 

We shall thus find ¢ and 7 such that 


OF |r[<td (6) and |r|<bdy-(0) 


We take 0 4 rT = —(¢- z”)/t° to ensure that (2°, t°) belongs to Y(c,,), and then the problem is 
reduced to finding ¢ such that 


OF |¢- 2°|<£d4-(¢) and |¢- 2°|<4dx-(¢). 
(9.132) 


Since by hypothesis z? ¢ A = A”, there is a point ¢° € A” such that C°- z° = 0. If 2° is in the 
interior of A’, then finding ¢ is easy: we have dx+(¢°)>d4+(¢°) > 0 = |¢°- Z| and can take € 
close to €°. If on the other hand ¢° € A”, we argue as follows. By the homogeneous interior 
cone condition, 


sup <sl°|ada(C)>786"l 
= 2 


for some positive constant y. On the other hand 


sup |¢-2°| = s|[C°llal|z" 2. 


IIC-¢°|l2<s||C° lla 


Given any positive 0, it is thus enough to choose ¢ such that e|2°|<dy to satisfy the first 
inequality in (9.132) for some ¢ close to <°, more precisely satisfying ¢ —¢ °l<s|¢ : for any 
given sufficiently small positive s. The second is then satisfied strictly when ¢ = ¢°, because A* 
is strongly contained in K. by Corollary 9.8.3, so that dg+(¢°) > d4+(¢°) = 0, and it must 
therefore also be satisfied for all ¢ satisfying |¢ — ¢°|<s|¢°| for all sufficiently small positive s. 
This completes the proof. 


9.8.8 A necessary differential condition for -closed functions 


It is well known that convex functions as well as plurisubharmonic functions of class C? can be 
characterized by differential conditions. Is the same true for &-closed functions? We shall first 
establish a necessary differential condition. 


Proposition 9.8.50 Suppose that f is an Y-closed function of class C* in some open set Q of 
C*"\ {0}. Then 


IS (fesen — 2fefax)Bydk|< D> feyabyde, in forallbe CM”. 


(9.133) 


In particular, ifn = 1 and we define F(z) = f(1, z), z € C, then 
Fee = 2F?|<Fz. 
(9.134) 


Proof Define g(z) = —log|G- z|. For every point a where f(a) is finite there is a vector 8 such 
that gradg(a) = gradf(a). Indeed, let us first note that by homogeneity S> aj; fz,(a) = —1/2 
for all a. If we choose 8; = f,,(a), then B- a = —1/2 and 


0 — 1 8B; 
oz (2) => ae 


takes the value 


at z = a. Then by &-closedness f(z)>f(a) + g(z) — g(a) for all z, for the definition of the Y- 
transformation uses precisely the functions g plus a constant. Take a curve t +> y(t) such that 
(0) =a and compare the two functions y = f*y and w= f(a)+ 9° y- g(a). We have 
y(0) = W(0) and y’(0) = w’(0) and must therefore have yp" (0)>"(0). We calculate 9’: 


=o" = Re S> ie 7, (t) 
+ - fea? y(t) +Re S f.,(7 


The corresponding formula for y simplifies to 


2" (t) = Re Yo gej(rt)) Ont) + Re Y) 92,1) 77); 


since gz, is holomorphic, i.e., 92,3, = 0. Also 


B;B 
Moreover gz, = fz, at z = a, so that 


7" (0) = 2Re SO fz,(a) f,(a )+Re S° f(a 
The inequality y"(0) >" (0) then means that 
Re S> fejze(a)7j(0)74(0) + S> fejae(a)7}(0)7,(0) + Re S > fe,(a)7/(0) 
>Re Y/ fz,(a) fax(a)74(0)74(0) + Re D | f.,(a)15 (0). 


If we now let the direction y'(0) vary, this means that (9.133) holds. 


We shall now prove that the differential condition (9.134) is not sufficient for &-closedness in a 
simply connected domain which is not a disk. 


Lemma 9.8.51 Let f be an &-closed function which is of class C’ in a neighborhood of a point 
ae Crt 0h. We consider its restriction to 29 = 1, and write z for the coordinate there, thus 
F(z) = f(1,z). Let G(z)=log|1+6z|, z€C. We now choose f such that 
OF (a)/dz = OG(a)/Oz = —B/(1+ Ba). With this value of B we have 
F(z)>F(a) + G(z) — G(a) for all z. In particular F(z)>F(a) at all points on the circle 
[1 + Bz| = |1+ Bal. 

The proof is easy. 


Proposition 9.8.52 Let @ be a connected open subset of the Riemann sphere S* = C U {oo} 
such that S*\@ has at least one component which is not a disk. Then there exists a function F 
which is +00 in S?\w, C'™ in «, and satisfies the differential condition (9.134) in w such that 
LLfAfF at some point in @ for the corresponding function f defined by 
f (20, 21) = F(z1/20) — log |zo]. 


Proof Let K be the complement of a component of $?\@ which is not a disk; thus K contains @. 
Moreover the complement of K is connected and 0K C Ow. Let a,b,c,d be the four points whose 
existence is guaranteed by Lemma 9.4.35; recall that b,d € K anda,c ¢ K. Since they are on a 
circle, we can move them by a Mobius transformation: we can move a to 0 and c to ~, and b,d to 
some points on the real axis, d < 0 < b. Also b,d € Ow since b, d € OK. Now define 


exp(—1/(Imz+e)) whenz€w, Rez>0, Imz> -—e, 
F(z) = <0 at other points in w, 
+00 otherwise. 


Here we choose ¢ > 0 so small that the disk ||z||2<e does not intersect K; this means that the 
function is continuous, even identically zero in a neighborhood of the intersection of w and the 
imaginary axis. Moreover we choose ¢ so small that F satisfies the differential condition 
(OF /Oy)’<0?F/dy", and such that the point b; = b —ie belongs to . This means that F 
satisfies (9.134). Now at a point b, near b,, F takes arbitrarily small positive values. If there is a 
function G tangent to Fat such a point, it forces F to be positive at some point with negative real 
part. Indeed, the circle |1 + 8z| = |1 + {b2| passes through points arbitrarily close to the line 
Im z = Im by (see Lemma 9.8.51). This is a contradiction, since we defined F to be identically 
zero at all points in @ in the left half plane, and there are such points by construction. 


(De 
9.9 Lineal Convexity in Infinite Dimension 


Abstract of this section 
The purpose of this section is to study some problems on lineal convexity in spaces of infinite 
dimension, in particular to prove that a pseudoconvex or lineally convex set which is open in a 


subspace can be extended to an open set in the whole space which is pseudoconvex or lineally 
convex, respectively. 


Given a pseudoconvex open set @ in ck regarded as a subspace CF x {0} of CK x C™*§ = ©”, 
we can fatten it into pseudoconvex set which is open in the whole space by taking a Cartesian 
product w x w’ for a suitable set w in C"—*. How can this be done in an infinite-dimensional 
space? It is also of interest to construct a set which tapers off at the boundary of @. 

In an infinite-dimensional Hausdorff topological vector space the subspaces of finite 
dimension (which are always closed) and those subspaces of finite codimension which are closed 
are of particular interest. Here the codimension of a subspace F of E is the dimension of the 
quotient space E'/F. 


9.9.1 Generalizing lineal convexity 


We start by generalizing the concept of lineally convex set to higher codimenions: 


Definition 9.9.1 Let E be a topological vector space and A a subset of E. We shall say that A is 
m-lineally convex if its complement is a union of closed affine subspaces of codimension m. 


Thus 1-lineal convexity is just lineal convexity as defined in Definition 9.4.1 on page 279. An 
m-lineally convex set is also k-lineally convex for k>m. An m-lineally convex open set in C” is 
(n — m)-pseudoconvex in the sense of Rothstein (1955:130). 


9.9.2 Inverse images of m-lineally convex sets 


The intersection of an m-lineally convex set with a subspace is also m-lineally convex in that 
subspace—more generally, we have the following result. 


Proposition 9.9.2 Let f : HE — F be a continuous linear mapping of a topological vector space E 
into another one, F. If Y is an m-lineally convex subset of F, then its inverse image A = f (Y) 
under f is also m-lineally convex. 

Proof Take x € E\X. Then y = f(x) belongs to F\Y and by hypothesis we can find a linear 
subspace K C F of codimension m such that y+ K does not meet Y. The inverse image 


H=f “(K ) is a vector subspace of E and E'/H is of finite dimension. For the codimensions we 
obtain 


codimz(H) = dim E/H = dim f+(E)/K< dim F/K = codimrK = m. 


So E\X is a union of closed affine subspaces of codimension at most m, thus X is m-lineally 
convex. 


9.9.3, Constructing thicker sets 


Theorem 9.9.3 Let E be a normed complex vector space, V an open cone in E which is m- 
lineally convex, and F a vector subspace of E of dimension m. Assume that VN F = F\{0}. If 
@ 1S a pseudoconvex open set in F, then 


QM= N (#+V) 
LeF\w 


is open and pseudoconvex in E, and QQ) F = w. 


Proof We have 
E\Q = (F\w) + (E\V), 
(9.135) 


so that £'\2 is the vector sum of two closed sets in E. Let B be the unit ball in E. Since F is of 
finite dimension and V contains F'\{0}, there is a positive number s such that 


(F\B)+sBcV; 
in view of the homogeneity of Fand V we have for all r > 0 
[F\(rs-1B)] +rB CV. 
(9.136) 


Let now z € E\2 with ||z||<r/s. There is, according to (9.135) a representation = y + z of x 
with y € F\w and z € E\V. It follows that ||y||<r/s, for if we have ||y|| > r/s, it would follow 
from (9.136) that y € F\(rs~*B). This proves that 


TBA C [(rs~1) a (F\w)| + (E\V). 


Since (rs~!B) M (F\w) is compact and the vector sum of a compact set and a closed set is 
closed, we have finally proved that (rB)\ is closed; consequently, since ris arbitrary, that 0 is 
open. 

In order to prove that 0 is pseudoconvex, we shall find, given any point y € E\Q, a 
continuous linear projection 7 : E + F such that 1‘ (w) contains Q but not y. 

So let y belong to E\2. We then have y ¢ x + V for some x € F'\w. By hypothesis there 
exists a closed vector subspace H of E of codimension m and such that y — x + H does not meet 
V. Lemma 9.9.4 below shows that y— a+ H = 4H and since F is of codimension m and 
F' 1 H = {0}, we have E = F'+ H, a direct sum. The projection z : FE — F with kernel H and 
image Fis continuous. Since y — x € H, we have m(y) = 1(x) = a € F; that is, y ¢ 7 (w). On 
the other hand, 7*(V) = F\{0}, which after a translation proves that m+(4+V) = F\{z}, 
hence 


m*(Q) C O(F\ {2}; gz € F\w) =w. 


Summing up, y ¢ 7 (w) and 7 (w) D m (a*() = Q, the conclusion we wanted. 


During the proof above we needed the following lemma. 


Lemma 9.9.4 Let V be an m-lineally convex open cone in a topological vector space E. Assume 
that V contains F\\{0}, where F is a vector subspace of E of dimension m. Then A\V is a union 
of closed vector subspaces of dimension m; in fact, every closed affine vector subspace of E 
which is contained in E\V and of codimension m contains the origin and is a linear subspace. 
Proof Let H be a subspce of E of codimension m and such that (a + H) ™ V = 0. We shall thus 
prove that a+ H = H. If we had F+H FE, there would exist ce ¢ FO AH, « 40. Then 
a+rze€a+HcE\V. But a+rz belongs to V for all sufficiently large r since 
xz € F\{0} C V. This contradiction shows that F + H = E; it follows that a can be represented 
asa=y+2z,wherey € F,z€ Handy=a—zc€ca+H. Hence 


y€ (y+ A)NF Cc (a+ A)n(Vu {0}) c {0}; 


in other words that 0 € a + H, which is a vector subspace of E. This proves the lemma and so 
completes the proof of Theorem 9.9.3. 


Let us note that we also have the next result, which is closely related to the theorem just proved. 


Theorem 9.9.5 Let E, F V and Q be as in Theorem 9.9.3. Assume that that w is m-lineally 
convex in F. Then Q is also m-lineally convex. 


Proof Let y € E\2. As in the proof of Theorem 9.9.3, there is an x € F'\w and a projection 
n: E — F such that 1(y) = 1(x) =x. According to Proposition 9.9.2, 1°(w) is m-lineally 
convex and it follwos that OQ, the intersection of all the sets nt (w) for the various choices of 7, is 
also m-lineally convex. 


9.9.4 Constructing convex cones 


Finally we shall indicate how it is possible to construct cones V that can serve in Theorems 9.9.3 
and 9.9.5. 


Proposition 9.9.6 Let E be a topological vector space and let Aj, j7 =1,...,p, be mj-lineally 


convex subsets of E. Then the intersection A= A,;U--:UAp, is m-lineally convex, where 
M=M +++: +My. 


Proof Take any point a € E\A. Then a € F\A, for every j and there exists a closed subspace F 


of E of codimension m; such that a + F’; does not meet Aj. The intersection F = Fi 1---0 Fy, 


is closed and of codimension at most equal to m and does not cut A. We are ready. 


Let us denote by T the unit circle T = {t € C; |t] = 1}, and by TA = {ta; x € A} the circled 
set generated by a set A in a vector space. 


Lemma 9.9.7 If V is an open convex cone in a complex vector space E, then TV is 1-lineally 
convex. 


Proof If TV is equal to EF there is nothing to prove, so let us suppose that TV # EF. If 
a € E\TV, let us denote by F' = Ca the vector subspace generated by a. Then F does not meet 
V (note that the origin does not belong to V), so the Hahn—Banach theorem gives us a closed 


hyperplane H that contains F and does not meet V. Now TH = H, so H does not meet TV 
either, and, since a € H, this proves that TV is 1-lineally convex. 


Proposition 9.9.8 Let Vi,..., Vm be open convex cones in a topological vector space E. Then 
V=TYV,U-:-UTV,, 


1s m-lineally convex. 
Proof We combine Proposition 9.9.2 and Lemma 9.9.7. 
Proposition 9.9.9 Let E be a normed complex vector space, F a vector subspace of E and 


€1,---,&m nonzero linear forms on E such that their restrictions to F have norm at most equal to 
1. Then 


V = {x € E; |x — yl| < supja,,. m|Ei(y)| forsomey € F} 
(9.137) 


is an open m-lineally convex cone in E. 


Proof Let us define 
u,(z) = inf(|le —yl|—Reégj(y)),  2¢E, j=l,...,m. 
This is a convex function and it never takes the value —oo, since 
lz — yl| — Re ;(y)2||z — yll — llyll> — llzl| > —co 
for every  € EF and every y € F. Hence we have —||z||<u;<||x||. The cone 
V; = {a € E; u(x) < 0} 


is convex and open in view of the continuity of u,; From Proposition 9.9.8 we see that 
V=TY, U---U TV, is m-lineally convex. 


Let us also note that the cone V defined by (9.137) contains the cone 
m 
Vo = F\ a ker €;. 
jJ= 


This cone is open in F but not in E. If we make it a bit thicker to obtain an open cone in &, it can 
serve in Theorems 9.9.3 and 9.9.5 provided Fis a subspace of dimension m and 


Foker€,9---Nkerém = {0}; 


this latter condition means that the restrictions of the Gj to F form a base in the dual of F. 
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10.1 Introduction 


In the complex analysis of one variable, the Cauchy integral formula plays a 
central role. Indeed, virtually all of the fundamental results in the subject are 
derived from that simple formula. 


In the realm of harmonic functions, the Poisson kernel and integral formula 
play a central role. Fundamental results, such as the Fatou theorem about 
nontangential boundary behavior of harmonic and holomorphic functions, 
depend critically on the nature of the Poisson kernel, and particularly on the 
shape of its singularity. 

In the complex analysis of several variables—where the quality of the 
analysis depends on the geometry of the domain in question—the nature of 
kernels plays an even more critical role. On the one hand, it is generally very 
difficult to calculate the relevant kernels. On the other hand, knowledge of the 
kernels is a valuable tool. 

It is a pleasure to thank Joseph A. Cima for a careful reading of this paper 
and for many constructive suggestions. 


fe —_____ 


10.2 The Cauchy Formula in One Complex Variable 


Throughout this paper we use the term domain to mean a connected open set in C 


or C”. 

Viewed properly, the Cauchy integral formula in one complex variable is not 
a result about holomorphic functions. Rather, it is a result about arbitrary smooth 
functions. And it is a corollary of Stokes's theorem. We now explicate this point 
of view. 

The language of differential forms is needed in order to formulate Stokes's 
theorem. We say just a few words about the matter here. We shall only need 
differential 1-forms. You know from calculus that, in the plane, these are 
generated by dx and dy. In complex analysis, we let 


dz = dz +idy and dz = dx — idy. 


Correspondingly, we define 


and 
O05 _ 0, _ 
x2 =1, 52 = 0. 


There are corresponding intuitively appealing pairings between 0/0Oz and dz and 
between 0/02 and dz. 
If w = udz + vdz is a 1-form, then 


dw = SdzAdz+ &dzAdz= (% — S)dz Adz. 


(10.1) 
Of course, 1-forms can be integrated over 1-manifolds, such as the circle, and 2- 
forms can be integrated over 2-manifolds, such as the sphere or torus or a planar 


domain. 
Finally, we recall the exterior differential operator d. In the complex setting 


we can decompose d = 0 + 0, where 
Ow = & dz A dz 


and 


Refer back to equation (10.1). 


Theorem 10.2.2. (Stokes) Let 2 C C be a bounded open set with C! boundary. 
Let w be a differential form of degree 1 with coefficients inC1(Q). Then 


i w= f dw= f au + Bu 
an Q Q 


Standard references for Stokes's theorem are [RUD] and [DER]. The Stokes's 
theorem that we have recorded here is the standard one, simply expressed in 
complex notation. 

Now we have 


Proposition 10.2.3 Let 2 C C be a simply connected domain with C! boundary. 
Let f be a C' function on 22. Then it holds, for z € 2, that 


f= f re ac com fA dg A de. 


(10.2) 
Proof: Fix a point z € 92 and let € > 0 be so small that D(z, €) C 2. We apply 


Stokes's theorem to the 1-form f(¢)/(¢ — z) d¢ on the domain D, = Q\D(z,€). 
Thus we obtain 


ie ake hf I Raw, 


(10.3) 


Now, as € — 0°, the right-hand side of equation (10.3) tends to 


Of /d 
am | J BI ae n ae, 


just because 1/(¢ — z) is integrable. Let us examine the left-hand side of (10.3). 


We write 
A [4 mo 


a -af[ a rs ne d¢. 


aD(0,<.) § — 2 


It is worth noting that OQ and OD(z,€) have opposite orientations because of 
their different roles in the boundary of 2,. Now the first expression on the right- 
hand side of this last display is the first term of the right-hand side of (10.2). It 
remains to analyze the second term on the right-hand side of this last display. 

In fact, we have 


Ect 


x | f(¢) d= af f(z+ ee) ceit dt 
dD(0,€) 0 
= zf” f(zt+ ee") dtu 


> f(z) 


ase —> Or. 
Putting together all the pieces, we find that we have derived equation (10.2). 
It is important to note now that, if 0 is as in the statement of the o 


proposition and f € C+(2) and holomorphic on the interior, then equation 
(10.2) simplifies to 


af © 
f(z) = a [ Se. 


This, of course, is the Cauchy integral formula that one can find in any complex 
analysis text. 


10.3 Harmonic Functions and the Poisson Kernel 


If we take it on faith that the Dirichlet problem on the disc always has a solution 
(see, for instance, [GRK]), then we may discover a formula for that solution in 
the following fashion. 

Let fbe an L? function on OD, the boundary of the unit disc. Then we may 
expand f, in the L* topology, as a Fourier series: 


(ee) 


i yr oe a 
= a f(tje dt - e” 
EE. [ 
an ee oe 
= / no (2 > as) dt . 
0 


j=—00 


Fle®) = > Fine'* 


Now each character e’* has a harmonic extension to the disc given by 


yj(re*) = rJle%'. So it is natural to consider that the harmonic extension of fto 
the disc Dis given by 


27 oO 
fret) = f no( 2 2 nese) z 


j=—00 
It behooves us then to calculate the sum 
oe . 
P,(e(9)) — 3 S> lil et5(8—t) : 
j=—00 


Now we have 


~1 ee) 
P(e) = sore) +e So ret) f+ =I+II+I1II. 
—oo 1 
We see that 
Igy ewes 
i 
= by tertinies 
0 


j=0 
sh de i(9—t) 1 
= 9, 7e 1—re-*(9-4) 
= ie 
~ On ei(O-t)_p 
A similar calculation shows that 
Il= i—2 


As a result, 


P(e) = 1. oe +2.78-42=2 bei 


e—i(0-t)_p Qn Qn 1—2rcos(6—t)+r? * 


This is the Poisson kernel. 
The basic result about this kernel is as follows. 


Theorem 10.3.1 Let f be a continuous function on the unit circle T = 0D. Then 
the function 


F(re®) = P,* f(e). 


is continuous on D and harmonic on D. Note that F(e"’) = f(e’). 


ee 
10.4 Introduction to the Bergman Theory 


Stefan Bergman introduced his reproducing kernel in 1921. It was anticipated 
somewhat by the thesis of S. Bochner, which was later published in [BOC]. Since 
Hilbert space was a very new idea at the time, Bergman's approach made quite an 
impact. 

In this section we work only in C, the complex plane. Let 22C C be a 
domain. Define the Bergman space 


A?(2) = { f holomorphic on?: | io 
2 


24 A(z)" = IIfllaqa < oo} 


Observe that A?(2) is a closed subspace of L?(2). The orthogonal subspace B 
is also of interest. Of course, L?(Q) = A?(2) @ B. 


Lemma 10.4.1 Let kK C 22 be compact. There is a constant Cx > 0, depending 
on K, such that 


f(z) 


ue — C||F\|_ 42a) ,allfe A?(Q). 
ZzE 
Proof: In what follows, D(z, 1) is the open disc with center z and radius r. Also 
dA denotes 2-dimensional area measure. 

Since K is compact, there is an r(K)=r>0 so that, for any 
z € K, D(z,r) C 2. Therefore, for each z € K and f € A?(2), the mean value 
theorem tells us that 


4 1 
|f(z)| — AD(zn) 


/ fast 


(A(D(z,r))) 7 | fll zx ocr) 
<C-r|f lla 
=Ck\||fllaya. O 


IA 


Lemma 10.4.2 The space A?(2) is a Hilbert space with the inner product 


(f.9) = | fe@a@ ale 


Proof: Everything is clear except for completeness. Let { f;} C A? be a 
sequence that is Cauchy in norm. Since L* is complete there is an L? limit 
function f We need to see that fis holomorphic. But Lemma 4.1 yields that norm 
convergence implies normal (uniform on compact sets) convergence. And 
holomorphic functions are closed under normal limits. Therefore, f is 
holomorphic and A?(Q) is complete. o 


Lemma 10.4.3 For each fixed z € 92, the functional 
&,: fro f(z), f € A?(2) 


is a continuous linear functional on A?(22). 


Proof: This is immediate from Lemma 4.1 if we take K to be the singleton {z}. 5 


We may now apply the Riesz representation theorem to see that there is an 
element k, € A?(§2) such that the linear functional ®, is represented by inner 
product with k, : if f € A?(Q), then for all z € Q we have 


f(z) = &,(f) = ck kz). 


Definition 10.4.4 The Bergman kernel is the function K(z,¢) = kz(¢), z,¢ € 92. 
It has the reproducing property 


i) = / K(z,Q)f(0) dA(0), Vf € A2(0). 


Proposition 10.4.5 The Bergman kernel K(z,¢) is conjugate symmetric: 


K(z, o = K(G, z). 


Proof: By its very definition, K(¢,-) € A?(Q) for each fixed ¢. Therefore, the 
reproducing property of the Bergman kernel gives 


iE K(z,t)K(Gb dA(t) = K(G». 


On the other hand, 


= KZ Oe J 


Proposition 10.4.6 The Bergman kernel is uniquely determined by the properties 
that it is an element of A*(Q) in z, is conjugate symmetric and reproduces 
A?(Q). 


Proof: Let K'(z,¢) be another such kernel. Then 
Keo. = KCo= / K"(z,t)K(GB dA(t) 


= [ KGORE Haale 
=K'Z,9) =K'z,¢). O 
Since L?(2) is a separable Hilbert space, then so is its subspace A?({). 
Thus there is a complete orthonormal basis {bj} for A?(Q). 
Proposition 10.4.7 Let K be a compact subset of (92. Then the series 
SO) 


j=l 


sums uniformly on K x K to the Bergman kernel K(z, C). 


Proof: Let {¢;} be a complete orthonormal basis for the Bergman space. By the 
Riesz-Fischer and Riesz representation theorems, we obtain 


1/2 
sie) = mpltete2 ah 


o(S 


zek j=l 


= sup 


I {a,}l],2=1 
zek 


S> aj$;(z) 


j=l 


= sup |f(z) 
If ll42=1 
zek 


<Ck. 
(10.4) 


In the last lines we have used Lemma 4.1 and the Hahn-Banach theorem. 
Therefore, 


= oo W277 1/2 
d|#s(2)80) < (>: b;(z) ; (>: $;(C) ‘ 


and the convergence is uniform over z,¢ € K. For fixed z € 92, (10.4.) shows 
that {;(z)} i", € £*, Hence we have that > b;(z)b;(¢) € A?(2) as a function 


of ¢. Let the sum of the series be denoted by K'(z, ¢). Notice that K is conjugate 
symmetric by its very definition. Also, for f € A?({2), we have 


[RCO dA(Q) = 0 Fl 4s(-) = FO), 


where convergence is in the Hilbert space topology. [Here f( j) is the j” Fourier 
coefficient of f with respect to the basis {¢;}. ] But Hilbert space convergence 
dominates pointwise convergence (Lemma 4.1) so 


io = / K"(z,)#(Q)4A(O, all f € A2(0). 


Therefore, K is the Bergman kernel. O 


REMARK 10.4.8 It is worth noting explicitly that the proof of Proposition 4.7 
shows that 


Sn) 


equals the Bergman kernel K(z,¢) no matter what the choice of complete 
orthonormal basis {¢;} for A?(1). 


Proposition 10.4.9 If Q is a bounded domain in C then the mapping 


Pifis i. K(-,f(0) dA(0) 


is the Hilbert space orthogonal projection of L?(Q, dA) onto A?(Q). 


Proof: Notice that, by Fubini's theorem, Pis idempotent and self-adjoint and that 
A?() is precisely the set of elements of L* that are fixed by P. 

Given a holomorphic mapping ®: (2; — (22, we may ask to compare the 
complex derivative &’ with the real Jacobian determinant Jg(@). The fact of the 
matter, and this is easy to check just using the Cauchy—Riemann equations, is 
that 


det Jp(®) = |6"|?. 


A holomorphic mapping f : 2, — {2 of domains 2; C C, 22 C C is said 
to be biholomorphic (or conformal) if it is one-to-one, onto, and f’(z) ‘a 0 for 
every z € 921. 

In what follows we denote the Bergman kernel for a given domain 0 by Kg. 


Proposition 10.4.10 Let 921,922 be domains in C. Let f: 223 4 2 be 
conformal. Then 


f'(2)Ka,(f(2); AC) = Ko, (2; er 
Proof: Let ¢ € A?(2,). Then, by change of variable, 


3 f'(2)Ka,( Ff, FO) F'(Qd(¢) dA(¢) 


=i. f'(z)Ka,(f(2), OF (Ff 1(Q) o(f 10) 
x det Inf 1(¢) dA(Q). 


By the remark following Proposition 4.9, this simplifies to 


2) [| Kals@.O4 (FE7@)) 6(12@) $246. 


2, 


By change of variables, the expression in braces { } is an element of A?({22). So 
the reproducing property of Ky, applies, and the last line equals 


=//@Qs@2) oF Ge) =o@): 


By the uniqueness of the Bergman kernel, the proposition follows. Oo 


—1 


Proposition 10.4.11 For z € 92 it holds that K¢(z, z) > 0. 
Proof: Now 


If in fact K(z,z) = 0 for some z then ¢;(z) = 0 for all j hence f(z) = 0 for 
every f € A?(2) by Proposition 4.7. This is absurd. 


Definition 10.4.12 For any 92 C C, we define a Hermitian metric on 0 by 


g(z) = log K(z, z), zE9. 


This means that the square of the length of a tangent vector é at a point z € 2 is 
given by 


lel}. = g(2)EE. 


The metric that we have defined is called the Bergman metric. 


In a Hermitian metric g, the length of a C! curve y : [0,1] > 2 is given by 


(9) = / (a(-(t)) /(t) 7'(t)) at 


If PQare points of 0 then their distance de(P, Q) in the metric is defined to be 
the infimum of the lengths of all piecewise C! curves connecting the two points. 

It is not a priori obvious that the Bergman metric for a bounded domain (1) is 
given by a positive definite matrix at each point. 


Proposition 10.4.13 Let 21,2 CC be domains and Iet f : 21) — 2g be a 
biholomorphic mapping. Then f induces an isometry of Bergman metrics: 


flaz = |F'(2)§ 


for all z € 92,,€ € C. Equivalently, f induces an isometry of Bergman distances 
in the sense that 


B,f(z) 


do,(f(P); f(Q)) = da, (P,Q) 
for all P,Q © 224. 


Proof: This is a formal exercise, but we include it for completeness: 
From the definitions, it suffices to check that 


9 (f(2) (fw) (Fw) = 9% (zw 


(10.5) 
for all z € 2,w € C. But, by Proposition 4.10, 
g7 (2). = aoa log K 9, (2, 2) 
= slog {|f'(z)|?Ko,(f(2), f(2))} 
= slog Ko, (f(z), f(2)) 
(10.6) 


since log | f’(z)|? is locally 


log(f’) + log(7’) +C 


hence is annihilated by the mixed second derivative. But line (10.6) is nothing 
other than 


and (10.5) follows. Oo 


Proposition 10.4.14 Let 2. CC C bea domain. Let z © 22. Then 


K(z,z) = sup fey 


= 2 
Mle: en 
feA?(Q) "0A II Fl a2(qy=1 


f(z) 


Proof: Now 


K(z, z) = °\¢;(2)/? 
-( sup Teste 


{es }|e=1 


= sup ze 
I Ffll2=1 


f(z) 


by the Riesz-Fischer theorem, 


agp OF. 
peg le 


10.4.1 Calculating the Bergman Kernel 


We present three methods for calculating the Bergman kernel on the unit disc 
D CC. For the first, we use Proposition 4.7. 


Construction of the Bergman Kernel with an Orthonormal Basis 


Now we first notice that {¢ 2 a forms an orthogonal basis for A?(D). This 
follows just from circular symmetry. In order to use 4.7, we need an orthonormal 


basis. So we calculate that 


27 
[ie 2dA(C y= [Pf ro. rdrd@ = ay - 


Hence a complete orthonormal system for A*(D) is 


{ae Vf 2542-05 De 
V2n j= 0. 


C4 


We then may calculate that 


Set a = z¢. Then we have 


Kp(z,0) = 3 >» 23 + 2)a! 


4 |Q 
T 
Q 
— 


I 
ae ale 
. : 4 |2 
Qe. 
ret. oJ 
ie) 
&. 
Ne 


This is our first calculation of the Bergman kernel for D. 


Construction of the Bergman Kernel by Way of Differential Equations 


It is actually possible to obtain the Bergman kernel of a domain in the plane 
from the Green's function for that domain (see [KRA3, Section 1.3.3]). Let us 
now summarize the key ideas. Unlike the first Bergman kernel construction, the 


present one will work for any domain with C’ boundary. Thanks to work of 
Garabedian [GAR], one can say rather precisely what the Green's function of any 
planar domain is (see also [JAK]). 

First, the fundamental solution for the Laplacian in the plane is the function 


P(6,2) = dog |¢ - 2]. 


This means that A¢I"(¢, z) = 6, in the sense of distributions. [Observe that 6, 


denotes the Dirac “delta mass” at z and A cis the Laplacian in the ¢ variable.] In 
more prosaic terms, the condition is that 


/ CN ne ee) 


for any C™ function g with compact support. We write, as usual, ¢ = € +77. 
[This topic is treated in detail in [KRA1, Ch. 0, 1].] 

Given a domain §2 C C, the Green's function is posited to be a function 
G(¢, z) on 2 x 2 that satisfies 


G(¢, z) — IG z) F,(¢) ) 


where F',(¢) = F(C, z) is a particular harmonic function in the ¢ variable (to be 
specified momentarily). Moreover, it is mandated that G( - ,z) vanish on the 
boundary of 0. One constructs the function F'( - , z), for each fixed z, by solving 
a Dirichlet problem with boundary data I( - , z). Again, the reference [KRA1, p. 
40] has all the particulars. It is worth noting, and this point is not completely 
obvious but is discussed in [KRAI, Ch. 1], that the Green's function is a 
symmetric function of its arguments. 

The next proposition establishes a striking connection between the Bergman 
kernel and the classical Green's function. 


Proposition 10.4.15 Let QC C be a bounded domain with C* boundary. Let 
G(¢, z) be the Green's function for Q and let K(z,¢) be the Bergman kernel for 
Q. Then 


K(z,¢) =4- 5 G(G2)- 


(10.7) 


Proof: Our proof will use a version of Stokes's theorem written in the notation of 
complex variables. This matter was discussed earlier in the paper. It says that, if 


u € C!(2), then 


= 97% Ou 
ful ae = 2 [03 dé dn 


(10.8) 


where again ¢ = + %n. The reader is invited to convert this formula to an 
expression in ¢ and y and to confirm that the result coincides with the standard 
real-variable version of Stokes's theorem that can be found in any calculus book 
(see, e.g., [THO], [BLK]). 

Now we already know that 


G6, 2) = dog(¢ - 2) + Llog(C—%) + FG, 2). 
(10.9) 


Here we think of the logarithm as a multivalued holomorphic function; after we 
take a derivative, the ambiguity (which comes from an additive multiple of 277) 
goes away. 

Differentiating with respect to z (and using subscripts to denote derivatives), 
we find that 


G.(¢, z) = 4 = =P FAG, z) . 


We may rearrange this formula to read 


a = —4r-G,(¢,z) + 4 F.(¢, z). 


We know that G, as a function of ¢ vanishes on 092. Hence so does G,. Let 


f © C?(Q) be holomorphic on ©. It follows that the Cauchy formula 


_1f £9 
f(a) = Oni $ ar 


can be rewritten as 


fz) =p F(C)FLG, 2) dC. 


Now we apply Stokes's theorem (in the complex form) to rewrite this last as 
fe) =4- f SHOP G.2 dean, 
Q 


where ¢ = €+ 7%». Since f is holomorphic and F is real-valued, we may 
conveniently write this last formula as 


fle) =4- f (QFe(G,2) df dn. 
Q 
Now formula (10.9) tells us that Fiz = Gz. Therefore, we have 
fe) = ff SOA, 2) ag an. 
Q 


(10.10) 


With a suitable limiting argument, we may extend this formula from functions f 
that are holomorphic and in C?(2) to functions in A?(). 

It is straightforward now to verify that 4G¢ satisfies the first three 
characterizing properties of the Bergman kernel, just by examining our 
construction. The crucial reproducing property is, of course, formula (10.10). 
Then it follows that 


K(z,0) =4- sa G(2)- 


That is the desired result. 


It is worth noting that the proposition we have just established gives a 
practical method for confirming the existence of the Bergman kernel—by relating 
it to the Green's function, whose existence is elementary. See [HAP1] for a 
version of these techniques in the several complex variables context. 

Now let us calculate. Of course, the Green's function of the unit disc Dis 


G(¢, z) = x_log |¢ — z| — ~-log|1 — ¢2Z|, 


as a glance at any classical complex analysis text will tell us (see, for example, 
[AHL] or [HIL]). Verify the defining properties of Green's function for yourself. 
With formula (10.7) in mind, we can make life a bit easier by writing 


G(¢,z) = q_log(¢ — z) + qlog(¢— z) 
— Z_log(1 — ¢Z) — delog (I — G) . 


Here we think of the expression on the right as the concatenation of four 
multivalued functions, in view of the ambiguity of the logarithm function. This 
ambiguity is irrelevant for us because the derivative of Green's function is still 
well defined (i.e., the derivative annihilates additive constants). 

Now we readily calculate that 


G1 1,1 
Oz «4m Gz 4n = 1-@2Z 


and 


Cn re | 
OCOz An (1—¢z)” : 


In conclusion, we may apply Proposition 4.15 to see that 


K(z, ¢) = = . ae . 


This result is consistent with that obtained in the other two calculations (see 
above and below). 


Construction of the Bergman Kernel by Way of Conformal Invariance 


Let D CC be the unit disc. First we notice that, if either f € A?(D) or 
f € A?(D), then 


Foy =2f fr aa. 


(10.11) 


This is the standard, two-dimensional area form of the mean-value property for 
holomorphic or harmonic functions. 

Of course, the constant function u(z) = 1 is in A*(D), so it is reproduced by 
integration against the Bergman kernel. Hence, for any w € D, 


w= [Seo = | FRIo0) aalo, 


=2f fx(w,Q aa. 


By (10.11), we may conclude that 
— = K(w,0) 


for any w € D. 
Now, for a € D fixed, consider the Mobius transformation 


h(z) = 2. 


1-—az 
We know that 
1-|a|? 
h'(z) = a $ 


We may thus apply Proposition 4.10 with ¢ = A to find that 


K(w,a) = h'(w) - K(h(w), h(a) -R'(a) 
ak K(h(w),0) - 4 


(1—aw)? 1—al? 
— 1 . 
—aw)* 


a|R 


This is our formula for the Bergman kernel. 


10.4.2 The Poincaré-Bergman Metric on the Disc 


If D C Cis the unit disc, z € D, then Definition 4.12 shows that 


{ 2)au|? [ie v3] 
Byz = = 


(1—|z/?)” es 


tu 
where the subscript B indicates that we are working in the Bergman metric. We 


now use this formula to derive an explicit expression for the Poincaré distance 
from 0 € Dtor +10 € D,0 < r < 1. Call this distance d(0,1r). Then 


ee int / ‘| y(t) 


7 isacurvein D, y(0)=0, y(1)=r+i0}. 


B,y(t) dt : 


Elementary comparisons show that, among curves of the form 
w(t) =t+iw(t),0 <t <1, the curve y(t) =tr+ 0 is the shortest in the 
Poincaré metric. Further elementary arguments show that a general curve of the 
form w(t) = v(t) + iw(t) is always longer than some corresponding curve of the 
form t + iw(t). We leave the details of these assertions to the reader. Thus 


Since rotations are conformal maps of the disc, we may next conclude that 


d(0, re”) = Flog ( utr). 


Finally, if wi, w2 are arbitrary, then the Mobius transformation 


Z—-W1 
1-—wiz 


Qi zr 


satisfies d(w 1) = 0, ¢(w2) = (w2 — w1)/(1 — Wyw2). Then Proposition 4.15 
yields that 


d(w1, we) = d(0 me | 


’ 1—w)u2 
_1 feel 
v2 \ 1-21 
oo 
We note in passing that the expression 


p(wi, w2) = |(wi — w2)/(1 — Wiwe2)| is called the pseudohyperbolic distance. 
It is also conformallly invariant, but it does not arise from integrating an 
infinitesimal metric (i.e. lengths of tangent vectors at a point). A fuller discussion 
of both the Poincaré metric and the pseudohyperbolic metric on the disc may be 
found in [GARN]. 


ee 
10.5 Preliminary Facts about Boundary Smoothness 


The Riemann mapping theorem is a powerful result. It allows us to transfer the 
function theory of one proper, simply connected domain in the plane to another. 
But for many applications we need to know something about the boundary 
regularity of a conformal mapping. Does the mapping extend univalently and 
bicontinuously to the boundary? Does it extend smoothly to the boundary? 

Curiously, the second of these questions was answered first—by Paul 
Painlevé in his thesis [PAI]. About 35 years later, Constantin Carath’ eodory 
[CAR] answered the first question (and it is the first question that is typically 
treated in textbooks, not the second—although [GRK] is one of the exceptions). 

For many purposes, continuity to the boundary is insufficient. In the recent 
work [APF], studies were made of the Bergman kernel on a variety of domains 
with different boundary geometries. The behavior of the kernel on a simply 
connected domain { was studied by considering a conformal mapping 
y : 92 — D and the corresponding transformation formula 


Ko(z,¢) = Kp(¢(z), 9(0)¢'(2)'(@) 


(see Proposition 4.10). In order to pass back and forth, it was essential to know 
that the expression |y’(z)| is bounded above and below: 


cr < |y"(2)| < ce. 


(10.12) 


Note also that a set of inequalities like (5.1) tells us that the amount of stretching 
that occurs under the map g is bounded above and below; this fact was also 
crucial in the results of [APF]. 

A result that implies the kind of estimates we have been discussing is the 
following 


Theorem 10.5.2 Let 92 C C be a bounded, simply connected domain in C with 
sufficiently smooth boundary. Then any conformal mapping yp : 22— D will 
extend univalently and continuously differentiably to Q, and the inverse mapping 


yy! will extend univalently and continuously differentiably to D. 


With this theorem in hand, we know that |y’| extends continuously to the 
compact set 2. Thus it is bounded. In other words, 


ly’(z)| <C forallze”. 


Likewise, (p+)'| extends continuously to the compact set D. So it is bounded. 
But then 
1 —lf,-1) 
eT | _ IG ) (2)| <¢, 
and hence! 


ley (2) = 4- 


In the statement of Theorem 5.2, we have intentionally been imprecise about 
what “‘sufficiently smooth” means, since that is one of the main points of the rest 
of this section. For the moment, we shall content ourselves with proving the next 
result (see [BEK]). First, some definitions. 

We shall require a brief discussion of the concept of boundary smoothness. 
We will offer two approaches to the question. 


Boundary Smoothness by Way of Calculus: We will only consider domains with 
finitely many boundary components. Each boundary component will be a simple 
closed curve—see Figure 10.1. 

If S' is the unit circle in the plane, parametrized by t +> e”, then we may 
think of each boundary curve y; as given by 


y,:S13C. 


Let k € {0,1,2,...}. We say that 02 is C* if 7; is never zero, each j, and the 
first k partial derivatives of the function Vj exist, each j, and each derivative is 
continuous. 


Boundary Smoothness by Way of a Defining Function: In geometric analysis it is 
frequently useful to think of a domain ( in space as the sublevel set of a function 
P = pq. For example, the unit disc is given by 


DSeecspi2) Sle 12.0). 
(10.13) 


It is a nice exercise with the implicit function theorem to see that any domain 
whose boundary consists of C! curves can be written as the sublevel set of some 
function p, as in equation (10.13). We usually demand that Vp # 0 on O2 in 
order to prevent degeneracies and, more particularly, so that Ve(P) gives a well- 
defined outward normal vector at each boundary point P. We say that 
Q={zEC: p(z) <0} has C boundary, k € {0,1,2,...}, if there is a 
defining function p that is C*. 


Implicit in this discussion is an important point that ought to be explicitly enunciated. Namely, if 21 and 


22 are domains with rou boundary and y : 92; — {2 is a conformal map, then there are constants c,C > 0 
such that c - dist(p(z), 022) < dist(z,091) < C - dist(p(z), 02). In other words, g preserves— 
qualitatively speaking—the distance to the boundary. This assertion follows from the Hopf lemma, a result 
that we treat below. 


Co — 
O CP? 
bs, 


FIGURE 10.1 


A domain with finitely many boundary components. 


We say that a bounded domain 2 C C has C'™ (or smooth) boundary if there 
is a function p : C > R such that p is C™, 


2 = {zEC: p(z) < 0}, 


and V p is nowhere zero on O02. 
It is worth noting that possession of a defining function says nothing about 
the topology of the domain. The domain could be simply connected or not. 


Example 10.5.4 Let 


i {z€C: plz) = (22) 4 (22) "= iW of. 
Then 0 is the region bounded by an ellipse. Certainly, 0 has C’‘ boundary. 


Theorem 10.5.5 Letk € {0,1,2,...}. Then there is an integer N = N(k) such 
that, if QC C is a bounded domain with CN boundary, then any conformal 
mapping y : 2 — D will extend to be univalent and C* from 2 to D. Likewise, 


the inverse mapping y~' will extend to be univalent and Ck from D to 2. It may 
be shown—although it requires considerable extra effort—that N may be taken to 
bek+1, orevenk + €. 


This theorem is properly attributed to P. Painlevé, who proved it in his thesis 
[PAI]. Over the years, it has been sharpened through work of Kellogg, 
Warschawski, Pommerenke, and others. We shall indicate some of these results 
in what follows. 

Let (2 C C be a domain. We say that © satisfies Bell's Condition R if, for 
each k € {0,1,2,...}, there is an €€ {0,1,2,...} such that, whenever 
f:22—C has bounded derivative up to order {, then Pof (the Bergman 
projection of f) has bounded derivatives up to order k. Condition R has proved to 
be of historical importance in the study of holomorphic mappings. In this section 
we prove that D = D(0,1) satisfies a version of Condition R (see Theorem 
5.13). This will be the key to the proof below of Theorem 5.5. 

If Q is a domain and @: (2 > C, then we say that ® is k times boundedly 
continuously differentiable, and we write 6 € C}({), if all partial derivatives 


(0/dx)*(A/dy)'S of order s + t < k on Q (the interior of Q, not the boundary) 
exist, are continuous, and are bounded. We let Cp°(2) = ,C#(Q). If 
& & CF(2), then we set 


Ss t . 
Iles = Sy supl|()"(&) 2+) 
st+t<k 2 


We also need an idea of continuity of derivatives at the boundary: A function 
& : 2 > C is called C* if all partial derivatives of ® up to and including order k 
extend continuously to 2. As before, C() is defined to be NpC*(22). 


Definition 10.5.6 If f, g € C*(2), then we say that fand gagree up to orderk on 
OD if 


an = 0 


(£)'(2) 4 - a9) 


|e 


foralls +t<k. 


Lemma 10.5.7 For every k = 0,1,2,... there is a function d; € C*(R) such 
that 


1. Az (ax) =Oifz < $ 

2-Ay(e)— lite > 2, 

3.0 < Ax (x) <1 foralla ER 
Proof: Let 


Obviously w € C*(R). Define 
#(2) = (ze — 4) -v(-2+ 4). 


Then ¢ € C* and $(x) # 0 only if 1/3 < x < 2/3. Observe that 4(x) > 0 for 
all x. Let 


ia ij ; $(t) dt. 


Then u € C*(R), u(x) = 0 for z < 1/3, and u =c a positive constant for 
xz > 2/3. Define 


This function has all the required properties. o 


It is actually possible to find a C'™ function with properties 1, 2, and 3 of 
Lemma 5.7. 


Lemma 10.5.8 If 2CC is a domain with C boundary, 
Q={zEC: p(z) <0}, and if U is a neighborhood of 092 on which 


|Ve| > ¢ > 0, then there is a C function a, on Q such that 


lia, =00n QU, 
2. a, = 1 ina neighborhood of 022. 


Proof: Choose a number € > 0 such that, if —€ < p(z) < 0, thenz € UN 2. Let 
A, be as in the last lemma. Define 


Then a, has all the desired properties. Oo 


Lemma 10.5.9 Let 2 = {z © C: p(z) < 0} be a bounded domain with C~ 
boundary. Given g € C®(Q2), choose f(z) = p(z)g(z). If h € A?(Q) (the 
Bergman space—see Section 4), then 


[ FO S110) adn = 0. 


Proof: We want to integrate by parts, but A is not defined on O72 so we need a 
limiting argument. 

Let €>0 be small and let Q,={z:p(z)<-—e}. Then define 
fe(z) = (p(z) + €)g(z). Notice that fe€C™(Q<) and felan, =0. Also 


h € C®(,). So, by Green's theorem (or just the one-variable fundamental 
theorem of calculus, applied one variable at a time), 


[ ROR to) a dy = — f (gk) fel) d& dn, 


€ 


(10.14) 


There is no boundary term since h- fe|ag, = 0. But h € A?(2) so (0/O¢)h = 0 
and the last expression is 0. 
Finally, let 


1 if CEN, 
xel0) = {5 if Cg Q. 


Then, using (10.14), 
[| HOH) agar] =| [ HH) dean — | HO xe(6)-& tel 6) dan 
<[ -” AG lickin 


* fF 


By the Cauchy—Schwarz inequality, this is 


xel0)|-| EO - Ol agen, 


(a) ()-scebaa) 
(flan) ( 9 3) 3 (¢) “atan) ie 


Now | |1 — x<(¢)|? dé dn clearly tends to zero as € + 0*. Also we see that 


2 
Of./0¢ — Of /OC uniformly. So, letting € 0, we obtain 


[FOEO wean =0. 
Q 


That is the desired conclusion. Oo 


If  : 22) Mz is conformal, then we might expect the Bergman kernel for 
$2; to be related to that for (22. This is indeed the case, as our Proposition 4.10 
showed. We review it now: 


Theorem 10.5.10 If % : 22) — {, is conformal then 


Ko,(z,w) = $'(z)Ka,(®(z), P(w))P'(w). 


We conclude this section with Bell's projection formula, which is based on 
Theorem 5.10. 

If (2C C is a domain and K its Bergman kernel, then for any square 
integrable fon 0 we define 


Pof(2) = ip H(QKal2,¢) dé dn. 


This is the Bergman projection. 
Recall that, if O is a bounded domain in C, then the mapping 


pees [ K(-,0f(0) dA(0) 


is the Hilbert space orthogonal projection of L?(2, dA) onto A?(). 
Next is Bell's formula: 


Proposition 10.5.11 Let 2,, 22 be domains and let 
®: 123 S Do 
be a conformal map. Let f € L?(Q,). Then 
Po, (8) (fF 87) = (G7) - (Pa f) 7). 


Proof: We sketch the proof. For any f € L?(,) and g € L?(2), 


(Po,((B4)' (f° O)),9) = (84)! (fF), Pog), 
= (f, ((Pa,9)°®) - 8"), 
a (Po, f, ((Pa,g) &) : P'), 


= (((Pa,f)' 8) - (27), Pag), 
(((Pa,f) 4) (BY). 


Since g € L?(22) was arbitrary, we may conclude that Bell's identity is proved. 
o 


The next lemma, due to Bell, is central to the theory. 


Proposition 10.5.12 If 2 = {z © C: p(z) < 0} is a bounded domain with C~ 


boundary, if u € C**!(Q), and ifk > 1, then there is ag € C*(QQ) that agrees 
with u up to order k on O22 and such that Pong = 0. 


Proof: Let a, be as in Lemma 5.8. We define g by induction. For the C! case, let 


where 


bial ulo-ale) 
wi(z) = “Tapas 


Then 


nl) = arte) ule) + ole) & (HIG) 
= a4(z)- u(z) + p(z)-m(z), 
where 7/, is defined by this identity and is continuous on 1. Then 


v; —u= p(z)-m(z) near O02. 


(So v, and u agree to order zero on 022.) In particular, v1 — ulag = 0. Also 


Pne= [ K(z, 0) 2wi(6) dédn = 0 


by Lemma 5.9. 


Suppose inductively that we have constructed wy_, and vy_, = ower such 


that vy_; agrees to order (€ — 1) — 1 with u on 02 and Pyvy_; = 0. We shall 
now construct a function wy of the form 


we = weit: p" 


(10.15) 


such that vg = Zur agrees with u up to order £— 1 on 022 and Pov, = 0. 
Let ag be as in Lemma 5.8 and define a differential operator J on 2 by 


Sue 2p 06 
I) = arse Re (He) 


Notice that 
Qpo(z)=1 
(10.16) 
when z € O92. We define 
ten ee 
(10.17) 


Then, with wy defined as in (10.15) and vy = Lowe, we have 


QZ (u-w) =D tu- QF (wer + 0,+p*) 
= 9 (u—ve-1) — 6658 (Dp) 
+ (terms that involve a factor of p). 


If z € O92, then (using (10.16), (10.17)) 
QD (u—v.—1)-— DE "(u — ve1) +0 =0. 


(10.18) 


Since any directional derivative at P € OM is a linear combination of 
Q =a(P) 3 +b(P) Fs and r=-b(P) ZF +a(P) FZ, 
we may re-express our task as follows: We need to see that 
(r)°D*(u— vw) aq = 0 


for all s+t=£-—1 (notice that the case s+t<£—1 follows from the 
inductive hypothesis and the explicit form of we in (10.15)). The case 
s =0,t = — 1 was treated in (10.18). If s > 1, then we write 


TeD*(u — vw) =7(7° 1 D*(u — v2)). 
(10.19) 


Since (s — 1) +t = £-— 2, the expression in parentheses is 0 on O92 by the 
inductive hypothesis. But z is a directional derivative tangent to 02 (because D 
is normal); hence (10.18) is 0. 

The induction is now complete, and vz has been constructed. We set vy = g. 
Then 


Paal2)= ff KalzCa(c)agdn = ff Kal2,6)-Sewi(¢) adn = 0 


by Lemma 5.9. Oo 


We shall use Bell's lemma (Proposition 5.10) twice. Our first use right now 
is on the disc. First, note the following two simple facts: 


(a) If Kpis the Bergman kernel for the disc, then 


(b) If u € C*(D) and if uvanishes to order k at OD (i.e., u agrees with the zero 
function to order k at OD), then there is a C' > 0 such that 


lu(z)| < C+ (1 —|zl)*. 


Theorem 10.5.13. ((ConditionRfor the disc]) If k > 1 and u € C*+?(D), then 
|| Poul cit(p) 00: 


Proof: Use Proposition 5.12 to find a function v € C**!(D) that agrees with u to 
order k + 1 on OD and such that Pv = 0. Then Pp(u — v) = Ppu, and u—v 
vanishes to order k+1 on OD. In particular, by observation (b) above, 


lu(¢) — v(¢)| < C- (1 —|¢|)"**. Then, for j < & — 1, we have 


< | (f+ 1)'(1 — |¢|) 7 *C- (1 — |¢|)** dé dn, 
D 


where we have used observation (a) above. This last integral is clearly bounded, 
independent of z. 


REMARK 10.5.14 Item (b) above actually holds on any bounded domain (2 with 


C* boundary: If uw € C*() vanishes to order k on OM, then there is a C > 0 
such that 


lu(z)| < C+ 65(z)*. 


Here, for z € 92, 


do(z) = inf 


we 


z—-w 
Of course, 5g could also be replaced here by any C\smooth defining function. 


This is immediate from the definitions and the Taylor expansion in the normal 
direction (in terms of powers of the defining function p). 


pe 


10.6 Smoothness to the Boundary of Conformal Mappings 


Let 2 = {z © C: p(z) < 0} be a bounded and simply connected domain with 
C'™ boundary. Let F’': D — 2 be a conformal mapping. We wish to show that 


the one-to-one continuation of F to D (provided by Theorem 5.2) is actually in 


C®(D). For this we need a few lemmas. The first is a classical result from the 
theory of partial differential equations due to Hopf. 


Lemma 10.6.1 (Hopf's lemma) Let U C C be smoothly bounded. Let u be a 
harmonic function on U, continuous on the closure U. Suppose that u assumes a 
local maximum value at P € OU. Let v be the unit outward normal vector to OU 
at P. Then the one-sided lower derivative Ou/Ov, defined to be 

Ou __ (P)—u(P-tv) 


Ou — Jim inf HP)—HP—) 
dv t—0+ t 


1S positive. 


Proof: It is convenient to make the following normalizations: Assume that u 
assumes the value 0 at P and is negative nearby and inside U; finally take the 
negative of our function so that u has a local minimum at P. 

Now, since U has smooth boundary, there is an internally tangent disc at P. 
After scaling, we may as well suppose that it is the unit disc and that P = 1 + 20. 
Thus we may restrict our positive function u, with the minimum value 0 at P = 1 
, to the closed unit disc. Note in particular that u(0) > 0. Set C = u(0) > 0. 

The Harnack inequality shows that u(r) > [(1 — r)/(1 + r)]u(0), hence 


u(1)—u(r) _ =u(r) u(0) _ -C C 
1-r ~ 1-r SS 1t+r — 14+r = ie has 
The desired inequality for the normal derivative of u now follows. Oo 


REMARK 10.6.2 It is worth noting that the definition of the derivative and the 
fact that P is a local maximum guarantee—just from first principles—that the 
indicated one-sided lower normal derivative will be nonnegative. The Hopf 
lemma asserts that this derivative is actually positive. 


Lemma 10.6.3. If Q is a bounded, simply connected domain with C'~ boundary 
and if F : D — 92 1s a biholomorphic mapping, then there is a constant C' > 0 
such that 


5o(F(z)) <C(—|z|), allze@. 


Here 69(z) = ee 


—u 


Proof: The issue has to do only with points z near the boundary of D, and thus 
with points where F(z) is near the boundary of ©. Consider the function 
w ++ log |F'~!(w)|. This function is defined for all w sufficiently near 02 and 
indeed on 92\{F(0)}. And it is harmonic there. Moreover, it is continuous on 
(2Q\{F(0)}) UO, with value 0 on O2. In particular, it attains a (global) 
maximum at every point of 02, since |F~'(w)| < 1 if w € Q\{F(0)}. So the 
Hopf lemma applies. The logarithm function has nonzero derivative at all points. 
The conclusion of the lemma follows from combining this fact with the “normal 
derivative” conclusion of the Hopf lemma. That is, we show that 


2|F\|p>e>0 


at each P € OD. [Note that here, as in the Hopf lemma, no differentiability at 
boundary points is assumed: The derivative estimates are on the “lower 
derivative” only, which, as a lim inf, always exists and has value in the extended 
reals. | oO 


Lemma 10.6.4 With F,, 2 as above andk € {0,1,2,...} it holds that 
k —k 
(2) F()| < Ce - a). 


Proof: Since F takes values in 2, it follows that F is bounded. Now apply the 
Cauchy estimates on D(z, (1 — |z|)). O 


Lemma 10.6.5 If wy ©€ C?**+?(Q) vanishes to order 2k+1 on O09, then 


F'. (°F) € C}(D). That is, F'- (°F) has bounded derivatives up to and 
including order k. 


Proof: For 7 < k we have 


(2) (F'- wr) = 


But 


(2)? “wF) 
(10.20) 


is a linear combination, with complex coefficients, of terms of the form 
|((2)"¥) F@)] -(2)" Fe) (£)"F) 
where m < 7—@andn, +---+n,z <7 — &. So (10.20) is dominated by 


Ci 6p( F(z) 994 — [zy 0 — [| 
oC 5o(F(z))* oe aye’ 


By Lemma 6.3, this is 
<C-(1—|a|)**-(1=|z|) ** <¢. 


Oo 


REMARK 10.6.6 There is a remarkable formula of Faa di Bruno that formalizes 
the expansion for the higher derivatives of a composition. While the identity was 
first discovered in the eighteenth century, it is still being studied today (in the 
higher-dimensional version). 


Lemma 10.6.7 Let G : D > C be holomorphic and have the property that 
(£)’e)| < Cj < 00 
for j=0,...,k+1. Then each (0/8z)'G extends continuously to D for 


j=l,...,k. 


Proof: It is enough to treat the case k = 0. The general case follows inductively. 
If P € OD, then we define 


G(P) = / " GI(LP) - Pat + G(0). 


It is clear that this defines a continuous extension of Gto 0D. Og 


Theorem 10.6.8. ([Painlevé]) If 2 = {z © C: p(z) < 0} is a bounded, simply 
connected domain with C'~ boundary and F': D(0,1) > 2 is a conformal 
mapping, then F € C®(D) and F-! € C®(2). 

Proof: Let k € {1,2,...}. By Proposition 5.12 applied to the function u = 1 on 
9, there is a function v € C?**8(Q2) such that v agrees with u up to order 2k + 8 
on 02 and Pov = 0. Then ¢ = 1 — v satisfies Po(¢) = Pol — Pov =1 and @ 
vanishes to order 2k + 8 on 02. By Lemma 6.5, F’- (¢° F) € C#*3(D). By 
Lemma 6.7, F'.(¢° F) € C**?(D). By Theorem DAs: 
Pp(F'-(¢°F)) «© C*®1(D) and has k—1 bounded derivatives. But the 
transformation law (Proposition 5.11) tells us that 


Pp(F" - (¢° F)) = F": ((Pod)'F) = F!-1= F’. 


Thus F has bounded derivatives up to order k — 1. By Lemma 6.7, all derivatives 
of Fup to order k — 2 extend continuously to D. Since k was arbitrary, we may 
conclude that F ¢ C®(D). 

To show that F-! € C®(2), it is enough to show that the Jacobian 
determinant of F as a real mapping on D does not vanish at any boundary point 
of D (we already know it is everywhere nonzero on D). Since Fis holomorphic, F 
continues to satisfy the Cauchy—Riemann equations on D. So it is enough to 
check that, at each point of D\D, some first derivative of F is nonzero. This 
assertion follows from a Hopf lemma argument analogous to the proof of Lemma 
6.1. Details are left as an exercise. 


Classically, Theorem 6.8 was proved by studying Green's potentials. 
The result dates back to P. Painlevé's thesis. All the ideas in the proof we have 
presented here are due to S. Bell and E. Ligocka [BELL]. An account of Bell's 
approach, in a more general context, can be found in [BEK]. 


ee 
10.7 The Bochner—Martinelli Formula in Several Complex 
Variables 


If w is a differential form on Q C RN then, in coordinates, @ can be written as a 
finite sum of terms of the form w,dzx°, where a is a multi-index and @, is a 


smooth function. Differential forms on C” may be written in this fashion also, 
since C’ is canonically identified with R2”. However, it is much more convenient 
to use complex notation. Thus if 2 C C” and @ is a differential form on (2, then 


@ is a sum of terms of the form wagdz* A dz’, where a, 6 are multi-indices with 
la| <n, |B] <n. If0 < p,q <nand 


WwW = > Wap dz“dz°, 
|a|=p,|8|=¢ 


then @ is said to be a differential form of type (or bidegree) (p, q). 

In classical advanced calculus, only a differential form of total degree m may 
be integrated on a space or surface or manifold of (real) dimension m. Likewise, 
in our new notation, only forms of type (p, g) with p + q = m may be integrated 
on a space or surface or manifold of (real) dimension m. 


Ifw= S> Wagdz® /\ dz®, then we define 
a,B 


OW, 
0? dz; A dz® A dz, 
Oz; 


w= 
j=1 a6 


Letting d denote the usual exterior differential operator on forms (see [RUD], 
[FED]), we see by a straightforward calculation that d= 0+ 0. 
Notice that when fis a C! function (or (0,0) form) then 


Since the differentials dz; are linearly independent, we conclude that Of =O0on 
Q if and only if Of /0Z; = 0 for j = 1,...,n. That is, Of = 0 on 0 if and only if 
fis holomorphic in each variable separately. 

The language of differential forms is needed in order to formulate Stokes's 
theorem: 


Theorem 10.7.1. (Stokes) Let 2 C C” be a bounded open set with C! boundary. 
Let w be a differential form of bidegree (p,q) with coefficients in C+(Q). Then 


i w= f dw= f au + Bu 
an Q Q 


Standard references for Stokes's theorem are W. Rudin [1] and G. de Rham 
[1]. The Stokes's theorem that we have recorded here is the standard one simply 
expressed in complex notation. 


The full Cauchy integral formula in C is a formula not just about 
holomorphic functions but about all continuously differentiable functions. We 
now derive this more general result for all C”,n>1, and learn what 
consequences it has for the function theory of both one and several variables. 


Definition 10.7.2. On C” we let 


w(z) =dzAdzo/A...Adzpn 
n(z) = LS (—1)"""z,dzy (eee a dzj-1 /\ dz j+41 ORY A 6 i oe 
gal 


The form 7 is sometimes called the Leray form. We will often write w(Z) to mean 


dz he 2 WNdzZ, and likewise n(Z) to mean 
n 


Ss (—1)?""z,dz, IX Stief® dzj-1 /\ dz j41 Nise dda: 
j=l 
The genesis of the Leray form is explained by the following lemma. 


Lemma 10.7.3. For any z) € C”, any€ > 0, we have 


Te n(2) \ w(z) = toc w(Z) A w(z). 


Proof: Notice that dj(Z) = On(Z) = nw(Z). Therefore, by Stokes's theorem, 


Voce n(Z) \ w(z) = i d[n(Z) A w(z)). 


Of course, the expression in | | is saturated in dz’s so, in the decomposition 
d = 0+ 0, only the term @ will not die. Thus the last line equals 


/ [(O(n(Z))] Aw(z) =n / w(Z) Aw(z). 
J B(z°,e) J B(z°,e) 


Remark: 
Notice that, by change of variables, 


- w(Z) A w(z) = fore w(Z) A w(z) 
— @2n i w(Z) A w(z). 


A straightforward calculation shows that 


| tw(Z) A w(z) 
B(0,1) 
= (—1)%™) . (2%)" - (volume of the unit ball inC” ~ R2"), 


where q(n) = [n(n — 1)]/2. We denote the value of this integral by W(n). 


Theorem 10.7.4. (Bochner—Martinelli) Let 2. C C” be a bounded domain with 
C! boundary. Let f € C!(Q2). Then, for any z € 2, we have 


f(Qn(¢ — 2) Aw(C) 


an IC — 2|*" 


— to | EE ante —2) Awl) 


alae 


f(z) = Ww 


Proof: Fix z € §2. We apply Stokes's theorem to the form 


L{Q) = Ge 


on the domain 2,.= 2\B(z,€), where €>0 is chosen so small that 


B(z,€) C 2. Note that Stokes's theorem does not apply to forms that have a 
singularity; thus, we may not apply the theorem to L, on any domain that 
contains the point z in either its interior or its boundary. This observation helps to 
dictate the form of the domain 2,. As the proof develops, we shall see that it 
also helps to determine the outcome of our calculation. 

Notice that 


8(Q.<) = OQUOB(z,€) 


but that the two pieces are equipped with opposite orientations. 
Thus, by Stokes, 


=f (20). 
Qe 


(10.21) 
Notice that we consider z to be fixed and ¢ to be the variable. Now 
dcLi(¢) = O¢L.(¢) 
_ Of(QAn(C-2)Aw(Q) 
|¢-2/?" 
n ra) @ - = 
+ f(¢) —|{ oe }]w(¢) Aw(¢) 
» aC; ( |¢—2| ) 
(10.22) 


Observing that 


2( Cj 2) ) Slee 
ac, (Tea) = Tea ae 


we find that the second term on the far right of (10.22) dies and we have 


0 —Zz)Aw 


Substituting this identity into (10.21) yields 


op FQ ANE —2 Aw 
[ 20- Poco t= [ ae 


Z€ 


(10.23) 
Next we remark that 
= n(¢ — 2) Aw(C) 
Tes BACT) Vis C= 2" 
(F(¢) — F(z))n(¢ — Z) Aw(C) 

7 tec [Cale 

— T) + T>. 
(10.24) 


Since | f(¢) — f(z)| < C|¢ — z| (and since each term of n(¢ — Z) has a factor of 
some ¢;—2Z;) it follows that the integrand of T, is of size 


O(|C — 2|) °"** = €-2"*2, Since the surface over which the integration is 
performed has area ~ €2”~1, it follows that T2 > O ase — 0°. 
By Lemma 7.3, we also have 


T =€f(z) i n(C —2) \w(0) 


OB(z,€) 


= ne~" f(z) i . w(¢) A w(¢) 


(10.25) 


Finally, (10.23)—(10.24) yield that 


Oey, 


| n(¢—2) 
(Gai? 


=O(|¢-2z 


the last integral is absolutely convergent as € > 0* (remember that Of is 
bounded). Thus we finally have 


Fe) = satay f te0)— saber | 100 HGR AwlO. 


This is the Bochner—Martinelli formula. oO 


Corollary 10.7.5 If QC C is a bounded domain with C! boundary and if 
f € C*(Q) then, for any z € 2, 


f=% f Te a- 2 f (20/8) oe nae 


Proof: It is necessary only to note that, when n = 1, 


w(¢)=d¢, n(C —Z)=C-Z, and nW(n)=2z7i. 


Corollary 10.7.6 With hypotheses as in Corollary 7.5, and the additional 
assumption that 0 f = 0 on 2, we have 


ae oy ae) 
fle)= ae fp Pac, 


ee 


10.8 The Bergman Kernel in Several Complex Variables 


The basic theory of the Bergman kernel in several complex variables is exactly 
like that in one complex variable. The proofs are all soft analysis and proceed 
step by step in just the same way. 


All these ideas have been gathered under the rubric of “Hilbert space with 
reproducing kernel,” an idea first formulated by Aronszajn in [ARO]. All that is 
really needed to make the theory run is a version of our Lemma 4.1. For the 
Bergman space, the proof of this lemma in one variable works just as well in 
several variables. For the Szeg6 space, an application of the Bochner—Martinelli 
formula does the job. 


ee 
10.9 The Szeg6 and Poisson-Szeg6 Kernels 


The basic theory of the Szegé kernel is similar to that for the Bergman kernel— 
they are both special cases of a general theory of “Hilbert spaces with 
reproducing kernel” (see [ARO]). Thus we only outline the basic steps here, 
leaving details to the reader. 

Let QC C” be a bounded domain with C? boundary. Let A(Q) be those 
functions continuous on 2 that are holomorphic on 2. Let H?(02) be the space 
consisting of the closure in the L?(an, da) topology of the restrictions to 02 of 
elements of A({2) (here do is (2n —1)-dimensional area measure on the 
boundary). Then H?(9Q) is a proper Hilbert subspace of L?(02). Each element 
fceH (02) has a natural holomorphic extension to 0 given by its Poisson 
integral Pf. It is the case (see [KRAI, Ch. 8]) that, for o— almost every ¢ € 092, 
it holds that 


lim f(¢ - e%) = f(0). 
«07 
Here, as usual, v- is the unit outward normal vector to O02 at the point ¢. 
For each fixed z € §2 the functional 
wb,: H?(OQ)afH Pf(z) 


is continuous (why?). Let k,(¢) be the Hilbert space representative for the 
functional 7,. Define the Szegé kernel S(z, ¢) by the formula 


S(z,¢) =k.(¢) ,2€ 2, CE ON. 
If f € H?(O0) then 


Pf(2) = [ Se OF (Cael) 


for all z € (2. We shall not explicitly formulate and verify the various uniqueness 
and extremal properties for the Szegé kernel. The reader is invited to consider 
these topics. 

Let {;};", be an orthonormal basis for H (02). Define 


CO 


S'(z,0 = >> o(2)6(O , 46 € 2. 


j=l 


For convenience we tacitly identify here each function with its Poisson extension 
to the interior of the domain. Then, for K C §2 compact, the series defining S 
converges uniformly on K x K. By a Riesz-Fischer argument, S’(-,¢) is the 
Poisson integral of an element of H?(02), and S'(z,-) is the conjugate of the 
Poisson integral of an element of H7?(02). So S' extends to 
(Q x 2) U(Q x ), where it is understood that all functions on the boundary 
are defined only almost everywhere. The kernel S' is conjugate symmetric. Also, 
by Riesz-Fischer theory, S$ reproduces H?(02). Since the Szego kernel is 
unique, it follows that S = S’. 
The Szeg6 kernel may be thought of as representing a map 


S:fo |] f(OS(,Qdo(O 
an 


from L?(02) to H?(OQ). Since S is self-adjoint and idempotent, it is the Hilbert 
space projection of L?(02) to H?(02). 

The Poisson-Szeg6 kernel is obtained by a formal procedure from the Szegé 
kernel: this procedure manufactures a positive reproducing kernel from one that 
is not necessarily positive. Note in passing that, just as we argued for the 
Bergman kernel in the last section, $(z, z) is never 0 when z € 2. 


Proposition 10.9.1 Define 


P(z,0) = Sear ZEN, CEON. 


Then for any f € A(92) and z € 22 it holds that 


fla = ] FQ)A(z,¢)do(¢). 
On 


Proof: Fix z € and f € A({2) and define 


Then u € H?(02) hence 


f(z) 


| 
= 
eae 
| 
os 
~ 
R 
R 
= 
Ze 
S. 
ple 
ie’ 


This is the desired formula. oO 


REMARK 10.9.2 In passing to the Poisson-Szeg6 kernel, we gain the advantage 
of positivity of the kernel (for more on this circle of ideas, see [KRA1, Ch. 1, 8] 
and also [KAT, Ch. 1]). However, we lose something in that A(z, ¢) is no longer 
holomorphic in the z variable nor conjugate holomorphic in the ¢ variable. The 
literature on this kernel is rather sparse and there are many unresolved questions. 


As an exercise, use the paradigm of Proposition 9.1 to construct a positive 
kernel from the Cauchy kernel on the disc (be sure to first change notation in 
the usual Cauchy formula so that it is written in terms of arc length measure on 
the boundary). What familiar kernel results? 

Like the Bergman kernel, the Szegé and Poisson-Szeg6 kernels can almost 
never be explicitly computed. They can be calculated asymptotically in a number 
of important instances, however (see [FEF], [BOS]). We will give explicit 
formulas for these kernels on the ball. 


Lemma 10.9.3 The functions {z“}, where a ranges over multi-indices, are 
pairwise orthogonal and span H?(OB). 


Proof: The orthogonality follows from symmetry considerations. For the 
completeness, notice that it suffices to see that the span of {z*} is dense in A(B) 
in the uniform topology on the boundary. By the Stone-Weierstrass theorem, the 
closed algebra generated by {z°} and {Z°} is all of C(OB). But the monomials 


cae e a 0, are orthogonal to A(B) (use the power series expansion about the 
origin to see this). The claimed density follows. O 


Lemma 10.9.4 Let 1 = (1,0,...,0). Then 


Proof: We have that 


z~.1° ee oo 2h(k+n—1)! 
S(z, 1) = ead — = ee eS Se 
do 27 ll tacaB) 2 mk) * dy ki 
__ (n-1)! A (k+n—-1\ , 
~  Oan = & =i a 
(n—-1)! 1 


2nr (1—z)"° 


Lemma 10.9.5 Let p be a unitary rotation on C”. For any z € B,C € OB, we 
have that S(z,¢) = S(pz, pC). 


Proof: This is a standard change of variables argument, and we omit it. Oo 


Theorem 10.9.6 The Szeg6 kernel for the ball is 


S(2.0) = a 


1 
ae Sle) * 
Proof: Let z € B be arbitrary. Let p be the unique unitary rotation such that pz is 
a multiple of 1. Then, by 10.4, 


n—1)! 


ER ire = — 66.1) — 0! 
S(z, ¢) = S(p '1,¢) _ S(1, pC) aa S(p¢, 1) “Dan (Goay" 


(mh met 
Qn” (1-¢(e1)) 2m” (1-z¢) ~ 


Corollary 10.9.7 The Poisson-Szego6 kernel for the ball is 


P(z,C) = (m—1)! (1-|z]?)" 


ee 
10.10 A Variety of Different Kernels 


A kernel is constructible if it can be written down explicitly. A kernel is non- 
constructible (or canonical) if it is produced abstractly by some functional- 
analytic argument. 

In the theory of one complex variable, all canonical kernels (the Cauchy 
kernel, the Poisson kernel) are constructible, and all constructible kernels are 
canonical. 


Not so in several complex variables. Certainly in C” we may produce the 
Bergman and Szeg6 kernels abstractly. But, in general (except on domains with a 
great deal of symmetry like the ball or the polydisc or the bounded symmetric 
domains of Cartan), these kernels are quite difficult to construct. Fefferman 
[FEF] produced an asymptotic expansion for the Bergman kernel on a strongly 
pseudoconvex domain (and Boutet de Monvel/Sjéstrand [BOS] did something 
similar for the Szeg6 kernel), but these arguments are quite deep and difficult. 

It was considered to be quite exciting around 1970 when, independently, G. 
M. Henkin [HEN1], Grauert and Lieb [GRL], and Ramirez [RAM] were able to 
actually construct reproducing kernels on strongly pseudoconvex domains. These 
constructions were highly nontrivial, but they produced a formula quite 
analogous to the Cauchy kernel of one complex variable. In particular, these new 
reproducing kernels K(z,¢) had the important property that they were 
holomorphic in the free z variable. This meant that the new kernels K(z, C) 
would not only reproduce holomorphic functions; they would also create 
holomorphic functions. That is to say, if djs were any finite measure, then 


i) = / K(z, 0) du(é) 


will be a holomorphic function in z. 
The new kernels proved to be particularly useful. For instance, they could be 


used to produce explicit solutions to the equation 0u = f (see [HEN2], for 
instance, or [KRA1, Ch. 5]). They could also be used to attack various problems 
in the theory of function algebras. 

Here we sketch the Henkin construction for a domain ( which is strongly 
convex (all boundary curvatures positive). For the full details of the matter, 
consult [KRA1, Ch. 5]. 


We begin with an idea called the Cauchy—Fantappié formula or formalism. 
This is a device for creating reproducing formulas in complex analysis. Good 
references for this notion are [RAN] and [KOP]. 

Recall the notation w(z) and (z) from our discussion of the Bochner— 
Martinelli formula in Definition 7.2, ff. Further recall that a straightforward 
calculation shows that 


/ tw(Z) A w(z) 
B(O,1) 
= (—1)%) . (22)" - (volume of the unit ball inC” ~ R?), 


where q(n) = [n(n — 1)]/2. We denote the value of this integral by W(n). 


Theorem 10.10.1 Let QCCC” be a domain with C! boundary. Let 
w(z,C) = (wi(z,¢),---,Wn(z,¢)) be a C', vector-valued function on 


Nx A\ { diagonal } that satisfies 
S5 w3(z, OG; — 23) = 1. 
j=l 


Then we have, for any f € C1(2)/N { holomorphic functions on a} and any 
z € 92, the formula 


Fe) = soy f fOm(w) Aw(¢). 


Of course, it should be understood here that ¢ 1s the variable of integration and z 
1s a free variable. 


Sketch of Proof: For convenience we restrict attention to complex dimension 2. 
We may assume that z= 0 € 2. 


(a) Ifa? = (al, ad), a? = (a?, a2) are 2— tuples of C! functions on 9 that 


2 
satisfy S> ak (¢) -(¢; — 2;) = 1, let 
j=l 


B(a',a?) = S> E(a) a5.) 
JES 


\ O(a; (9) ’ 


where S, is the symmetric group on 2 letters (which, of course, has just two 
elements) and €(o) is the signature of the permutation o. Prove that B is 
independent of a!. 


(0) It follows that 9B = 0 on A\ 40 \ (indeed OB is an expression like B with 
the expression a} (1) replaced by 0a} o(1 1): 


(c) Use (b), ees the ahaa remark, to prove inductively that if 
B' = (bj, b3), B? = (b?, b3), then there is a form y on 2\{0} such that 


[B(a’, a”) — B(B", B)] Aw(¢) = dy = dy. 


(d) Prove that if at = a? = (wi, w2) then B(a', a”) simplifies as 


Blot, a”) A w(¢) = (2 — 1)!n(w) Aw). 


(e) Let Y be a small sphere of radius € > 0 centered at 0 such that Y C 92. 
Use part (c) to see that 


f(Qn(w) Aw(¢) = / f(Cn(w) A w(0). 
02 S 


(f) Now use (c) and (d) to see that 


i, f(Qn(w) Aw(0) = i f(On(vr) \w(0), 
of oH 


(2,0) = 3. 


where 


[Warning: Be careful if you decide to apply Stokes's theorem.] From the theory 
of the Bochner—Martinelli kernel, we know that the last line is 2- W(2) - f(0). o 


Now let 


2 = {zEC?: p(z) <0}. 


We assume that ( is strongly convex, 1.e., that the matrix of (real) second partial 
derivatives of p induces a positive definite quadratic form. 

Define the complex tangent line to 02 at a point P € OM to be the zero set of 
the function 


Lo(2) = oP) + gE (P)(zr — Pi) + GE (P)(za — Pa). 


Now the Taylor expansion of p about a point P € O92, expressed in complex 
notation, is 


p(z) = el 
+ 


P) + $2(P)(21 — Pi) + $2 (P)(z2 — Po) + 2 (P)(Z — Pi) 
S& (P)(Z2 — P2) + (quadratic terms) + &(z) . 


Here &(z) is an error term of order 3 and higher. 
This last may be rewritten (because P € 092 so p(P) = 0) as 


p(z) = 2Re Yp(z) + (quadratic terms) + &(z). 
And, because 2 is hypothesized to be strongly convex, we find that 


p(z) = 2Re #p(z) + (positive term) . 


We conclude that the variety {z : &p(z) = 0} only intersects 2 at the point 
P. For this reason, we may use the notation of the Cauchy—Fantappié formula to 
define 


0 Oz P 
wi(z,P) = eek 


and 


0 Oz2 P 
we(z, P) = earottn) ane : 


It follows then that 


So Cauchy—Fantappié theory applies, using this (w1,w2), and we get a 
reproducing kernel. 

In the case that © is the unit ball B, the reader may check that the resulting 
kernel is the Szeg6 kernel. 


i ee 
10.11 Constructible Kernels Versus Non-Constructible Kernels 


We close our discussion of reproducing kernels with a consideration of how the 
Szego kernel is related to the more explicit kernel that arises from the Cauchy— 
Fantappié formalism. This set of ideas is due to Kerzman and Stein [KES]. 

C. Fefferman [FEF] made an important contribution in 1974 when he 
produced an asymptotic expansion for the Bergman kernel of a strongly 
pseudoconvex domain. Basically he was able to write 


K(z,6) = P(z,¢) + &(z,¢); 


where P (the principal term) is, in suitable local coordinates, the Bergman kernel 
of the ball and & (the error term) is a term of strictly lower order (in some 
measurable sense). This powerful formula gives one a means for calculating 
mapping properties of the Bergman integral. Fefferman himself used the formula 
to calculate the boundary asymptotics of Bergman metric geodesics (for the 
purpose of proving the smooth boundary extension of biholomorphic mappings). 
Fefferman states in his paper—although the details have never been worked out 
—that there is a similar asymptotic expansion for the Szego kernel of a strongly 
pseudoconvex domain. 

At about the same time, Boutet de Monvel and Sjéstrand [BOS] used the 
technique of Fourier integral operators [HOR] to directly derive an asymptotic 
expansion for the Szegdé kernel of a strongly pseudoconvex domain. This 
expansion is quite similar to Fefferman's: there is a principal term, which in 
suitable local coordinates is the Szeg6 kernel of the ball, and there is an error 
term which is of lower order. It is not known whether the techniques of [BOS] 
can be used to derive an asymptotic expansion for the Bergman kernel. 

The main purpose of the present section is to consider another method, due to 
Kerzman and Stein, for deriving asymptotic expansions for the canonical kernels 
that is more elementary and uses less machinery. Fefferman's rather complicated 


argument uses Kohn's solution of the 0-Neumann problem as well as the theory 


of nonisotropic singular integrals. Boutet de Monvel and Sjéstrand's argument 
uses the theory of Fourier integral operators. The method of Kerzman and Stein 
[KES] that we treat here uses only basic complex function theory and a little 
functional analysis. 

At this time there are virtually no results about asymptotic expansions for the 
canonical kernels on weakly pseudoconvex domains. Some interesting partial 
results appear in [HAN]. 

The ideas that we present now have thus far only been developed on strongly 
pseudoconvex domains. It is an important open problem to determine how to 
carry out a similar program on finite type domains or more general domains. 

In previous’ sections, we have defined the Szegod projection 
S : L?(02) > H?(2). We also have a mapping H : L?(02) > H?(Q) that is 
determined by the constructive Henkin kernel discussed above. We note that H 
defines a bounded operator from L?(02) to H?(2) (the Hardy space—see 
[KRA, Chapter 8]) for the following reason. 

Let Q be strongly pseudoconvex. This means that the Levi form is positive 
definite at each boundary point (see [KRA1, Ch. 3] for details). It is known that 
O02, when equipped with balls coming from the complex structure and the usual 
boundary area measure (see [STE], [KRA1, Ch. 8]), is a space of homogeneous 
type in the sense of Coifman and Weiss [COW]. Further, it is straightforward to 
verify that the Henkin operator H satisfies the hypotheses of the David-Journé 71 
theorem for spaces of homogeneous type (see [CHR] for a nice exposition of 
these ideas). Thus we may conclude that the Henkin operator maps L?(02) to 
L?(0Q). Since the Henkin kernel also obviously maps L?(0Q) to holomorphic 
functions, we may conclude that the Henkin integral maps L?(02) to H?(09). 

Now of course S, being a projection, is self-adjoint. So S = S". It is not at all 
true that H = H’, but one may calculate (see below for the details) that 
A =H’ — His small in a measurable sense. 

We also have 


HS=S, SH’ =S, 
SH =H, H*S=H'. 


Let us discuss these four identities for a moment. 
For the first, notice that S is the projection onto H*, and H preserves 
holomorphic functions. So certainly HS = S. For the second, we calculate that 


(SH*z, y) = (H'z, Sy) = (x, HSy) = (x, Sy) 


(because H preserves holomorphic functions) and thus = (Sz,y). Hence 
SH’ = S. For the third, notice that H maps to the holomorphic functions and S 
preserves holomorphic functions. And, for the fourth, we calculate that 


(H’Sz, y) a (Sz, Hy) = (a, SHy) = (x, Hy) a (Hz, y). 


In conclusion, HS=-H. 
Now we see that 


SA = S(H* — H) = SH* -SH=S-H. 


As a result, 
S=H+SA 
SO 
S(I- A)=H. 
In conclusion, 
S—H(I- A)". 


If, indeed, we can show that A is norm small in a suitable sense, then 
(I— A) ~" is well defined by a Neumann series. Thus we may write 


S—-H+HA+HA?+..-4+HA!)+HAS1.... 


Hence we have expressed the Szeg6 projection S as an asymptotic expansion in 
terms of the Henkin projection H. By applying this asymptotic expansion to the 
Dirac delta mass, this last formula can be translated into saying that the Szeg6 
kernelS can be written as an asymptotic expansion in terms of the Henkin kernel. 

Some new ideas connected with the discussion here appear in [KRA4]. 

It should be noted that Ewa Ligocka [LIG] has shown that these same ideas 
may be applied to expand the Bergman kernel in an asymptotic expansion in 
terms of the Henkin kernel. We shall not treat the details of her argument here. 


10.12 Concluding Remarks 


Reproducing kernels are part of the bedrock of modern analysis. A detailed 
treatment of reproducing kernels in the real variable setting appears in [BIN1], 
[BIN2]. Of course, there are many books about integral kernels in the complex 
analysis setting. Certainly, [RAN] is one of the most comprehensive. The book 
[KRA1] also has considerable treatment of the Bergman, Szeg6, and Poisson- 
Szego kernels. 

We expect that the ideas presented here will be central to our subject for a 
good many years to come. 
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11.1 Preliminary 


The Riemann mapping theorem (Theorem 11.2.1) is the most important and 
beautiful theorem in the field of one complex variable. It basically asserts 


that any two reasonable simply connected domains in C are biholomorphic. 
This is a non-trivial result. We all know, if two domains are simply 
connected, then they are homeomorphic, i.e., a bijective continuous 
mapping from the first domain to the second domain exists. Indeed, for two 
reasonably simply connected domains, we can even find a bijective smooth 


mapping between the two domains. This result is classical. However, from 
smooth mapping to holomorphic function is not trivial at all. After all, a 
holomorphic function is a harmonic mapping (i.e., both components are 
harmonic). The generic smooth mapping is far away from being harmonic. 
This is the point of the Riemann mapping theorem which states, a 
biholomorphism (i.e., a bijective holomorphic mapping of which the inverse 
is also holomorphic) can still be found after a careful treatment. This note 
requires the reader to have a little background on complex analysis and 
partial differential equations. 


For the following, we give several useful and well-known theorems in 
complex variables. We also include a brief proof for some of them. For the 
details, readers are referred to standard textbooks of complex variables (see, 
e.g., Taylor [6]). 


The first theorem is called the argument principle. 


Theorem 11.1.1 (argument principle). Let C C C be a closed contour, 1.e., 
a simple closed Jordan curve. Let D be the region enclosed by C. If f(z) is 


a holomorphic function on D, and f has no zeros on C, then 


1 f'(z) — 
mi Jy flay NO) 


where N(0) denotes the number of zeros inside the contour C, with 
multiplicities counted. 


Here, we give a very brief proof. By change of variables z > w := f(z), 
we have that 


/ 
1 
1 Ie) G2= sf — dw = N(0), 
anv-1 J, f(z) Qnv-1 #(C) W 


where the last equation is due to a direct computation. This completes the 
proof of the argument principle. 


We are not satisfied by knowing how many zeros are inside a contour C. 
Roughly speaking, for a fixed g in the image of f, we want to compute the 


number of z so that f(z) = q. For this, we observe f(z) — q and obtain the 
following generalized argument principle. 


Corollary 11.1.1 (generalized argument principle) Let C' Cc C be a closed 
contour, 1.e., a simple closed Jordan curve. Let D be the region enclosed by 
C. Let f(z) be a holomorphic function on D. Fix q € f(D) and assume 
there is no z € C so that f(z) = q. Then 


_f 10. 
anv f. igug 


where N(q) denotes the number of z for which f(z) = q inside the contour 
C, with multiplicities counted. 


We observe that 


; f'(z) : 
anv-1 f. f(z) -4 ae 


f'(2) 


is continuous on C if 
f(z)-4@ 


for all holomorphic functions f Observe that 
and only if f(z) 4 q on C. Thus, 


1 f f '(z ) d 
————. dz 
tev Je f(z) —4q 
is continuous. But a continuous function with image of integers is a 
constant function. This completes the proof. Moreover, the generalized 


argument principle says N(q) remains constant on each component of 
f(D)\f(C) as long as no points in f~!(q) cross the boundary. 


The generalized argument principle is quite useful for determining the 
bijectivity of a mapping. We will see it soon. 


Theorem 11.1.2. (Strong maximal principle and Hopf lemma) Let U be a 
bounded domain with smooth (C™@) boundary and u be a non-constant 


harmonic function on U. Then there exists a point p€ OQ so that 


u(p) = max,_,;u(z) and u(p) > u(z) for arbitrary z ¢ U. Moreover, at p, 
we have that &“(p) >0, where S“(p) >0 is the outward directional 


On 
derivative at p. 
In the proof, we also need the following Schwarz reflection principle. 


Theorem 11.1.3. (Schwarz reflection principle). Assume that f : U > C, 
where U is an open subset of C and OU is real analytic, is a holomorphic 
function which extends continuously to OU. If f maps OU into another real 
analytic curve M CC, then f extends holomorphically to an open 


neighborhood of U. 


11.2 The Riemann Mapping Theorem and Its Proof 


Theorem 11.2.1. (Riemann mapping theorem). Let U be a non-empty 
simply connected open proper subset with real analytic boundaries OU of C, 
then there exists a biholomorphic mapping f (1.e. a bijective holomorphic 
mapping whose inverse 1s also holomorphic) from U onto the open unit disk 


D := {zEC: |z| < 1}. 


For simplicity, we will call such a mapping a Riemann mapping. 


In this note, we only consider bounded domains with real analytic 
boundaries. 


To construct a Riemann mapping, we assume that 


f(z) =(z— zy eu t+v-lv(2), 


for some Zp € U. Here u and vare real functions that will be defined later. 
Since the exponential function has no zeros, f(z) will have its only zero at 
Z. By the generalized argument principle, f must be bijective, because 
N(q) = N(0) for an arbitrary g € f(U). We want to construct f with some 
constraints as follows. 


1. The function fhas to be holomorphic. 
2. We hope the image of fto be the unit disc D. 
3. We need to verify that the fis a biholomorphism. 


Among others, we look at condition 2. Since we hope the image of fis the 
unit disc D, we hope f maps OU to OD. For this aim, we hope |f(z)| = 1, 
whenever z € OU. It is equivalent to say that z— z e“() = 1. We, hence, 
construct a function f as follows. We want to find a harmonic function u 
defined in U such that 


u(z) = — log |z — zol, 


for z € OU. 


By solving the Dirichlet problem, one can find a harmonic function such 
that 


u(z) = — log |z — zol, 


for z € OU. Indeed, Green's kernel extends to a simply connected domain 
containing U by the Schwarz reflection principle, and so uis harmonic on a 


larger domain containing U. After finding u, we need to find the conjugate 
harmonic function v. The vis defined as follows: 


z 
v= | Uz dy — Uy dz. 
20 


(11.1) 


Since U is simply connected and uzz = —uy, by the harmonicity of u, one 
can see the definition is independent of the path from Z to z. The u and v 


clearly satisfy the following Cauchy—Riemann equation: 
Ue =v, and lg Ug. 


We will use Theorem 11.1.2 to verify flay is a diffeomorphism from OU 
onto OD and then we can verify by the argument principle, the condition 3, 


namely the biholomorphism from U onto D. On OU, 


f(z) _ (z _ zoe 8 zz) +V-1v(z) __ _z—z0 ev-lv(z) = eV-1(Arg(z-20)+v(2)) 


~ |z—zo| 


Thus, for arbitrary z € OU, | f(z)| = 1,ie., f(z) € OD. 


We want to show that eV-2¥ := eV-HArs (220)+v(2)) is a covering map from 
OU onto OD. Observe that the conjugate harmonic function of v is 
—p := —log|z— zo| — u(z). Since, by the Cauchy—Riemann equation, the 
tangential directional derivative of v is the same as the outward directional 
derivative of u, we first verify ou ~ 0 on OU. By the strong maximal 


principle and the Hopf lemma, one can see that oe > 0 at all maximal 
points. Indeed, by ulay = —log|z— zo||ay, we see that w|ay = 0. And 
when z — Zo, / — —oo. So it turns that all points on OU are maximal 
points, and hence, we can use the Hopf lemma on all OU. It implies 
$e (z) > 0 for all z € OU. Thus, the tangential directional derivative of v is 


Wa 


nonzero and eY~” is a local diffeomorphism. So eV!” isa covering map. 


Since eV!” is a covering map, so does flay. 

At the last, we use the generalized argument principle to verify that fis a 
bijective holomorphic function from U onto D. Indeed, f|ay maps OU to 
OD and thus, if |g| < 1, there is no z € OU so that f(z) = gq. By inspecting 


f(z) =(z- zo )eul2)+V¥-1v(2) 


one finds Z) is the only zero of f and it is simple, namely N(0) = 1. 


Consequently, N(q) = 1 for all gq € D by the discussion after Corollary 
11.1.1. So fis a biholomorphism by Corollary 11.1.1. 


eS  HoHoHM,!,—Y_=__ ee 
11.3 An Example 


We now bring the Riemann mapping theorem to some precise examples to 
see what these Riemann mappings look like. To get a precise computation, 


we need a bounded domain of smooth boundary with an explicit Green's 
function. So we mainly work on the unit disk because otherwise, the Green 
function is hard to obtain explicitly. 


Example 11.3.1 Let Ube the unit disc D and zp € D. 


One can check that u(z) = — log |1 — z9z| solves the following Dirichlet 
problem: 


Au =0 zeD 
u(z) = —log|z— z0| z€ OD. 


Indeed, on OD, we have that 


|1 — z92|? 
= 1+ |z|?|zo|? — 2Rezoz 
= |z|* + |zo|? — 2Rezoz 
= |2— 26: 


It is also easy to check u is harmonic because the only singular point of 


2 a*o(l~2oz) _ zy 


J\l—zoz|2 ss Lez 


log |1 — zoz| is outside D and # log L 2267 
which is holomorphic. 


We are going to use u to look for v. Letting 2 = 29+ V-ly and 


z=x2+v-ly, we let g(z) = 1 — oz and thus, u(z) = — log |g(z)|. One 
can definitely use the knowledge of Calculus III (finding potential) to find 
the conjugate vof u. Here, we give another approach. 


Observe that g(z) maps the closed unit disk into the right half plane 


K = {w EC: Rew > O}. The principal branch 
Log(w) = log w + V-larg w of the logarithm with 5 < arg ur Fas 
holomorphic on K. Therefore, the composite function 


Logg(z) = log g(z) + V-largg(z) is holomorphic on D. Hence, arg 9(z) 
is the harmonic conjugate for log |g(z)|. Thus, 


v = arg(1 — 292). 


This gives us 


f(z) = (z _ zoe 1° 1—zoz +-V-larg(1-202) __ (z = zy )e°8 Far _ 220 


1—zoz° 


One can see this is actually a Mobius transformation that maps Zp to 0. 


For the cases of unbounded domains and rough boundary, readers may 
consult the book Taylor [5] (Page 398-402). Other references include 
Conway [1] (Page 277), Fuchs [2] (Page 31-32), Needham [3] (Page 550- 
554) and Nevanlinna—Paatero [4] (Page 325-327). 


Acknowledgments. The author thanks the anonymous referee for 
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12.1 Introduction 


In his work on automorphic functions [6] Fricke shows that if TC SL(2,C) is a subgroup, then the trace of any 
word in the generators is a polynomial with integral coefficients in the finitely many variables consisting of the 
traces of the generators of I’ together with finitely many of their products. The case I has two generators, 
I = (A, B) has been particulary well studied. There the trace of any word w(A, B) € I is a polynomial in the 
three complex variables x, y and z where 


x=tr(A), y=tr(B), z= tr(AB). 


(12:1) 


As a consequence every conjugacy class of an element w(A, B) in I uniquely determines a polynomial P(z, y, z) 
with integral coefficients: define P by 


P(a, y, z) = tr(w(A, B)). 


Horowitz [17] showed that the polynomial P may not determine the conjugacy class of w(A,B) uniquely, 
although, for a given polynomial, there are only finitely many conjugacy classes represented by cyclically reduced 
words w(A, B) giving P. These results are largely based around the conjugacy invariance of trace and Fricke's 
simple identity 


tr(AB) + tr(AB“) = tr(A) - tr(B). 
Traina [29, Corollary 1.] develops a family of trace identities to establish the following theorem. 


Theorem 12.1.1. (Uniqueness) Cyclically reduced words w; = w (a,b) and wz = w2(a,b) can have the same 
trace polynomial only if the absolute values of the exponents of the generators of a in wz arise from those inw , by 
a permutation, and the same must be true for the exponents of b 


Our initial interest lies in understanding these polynomial trace identities further and their connection with 
discrete groups of Mobius transformations. The Mobius group acts as linear fractional transformations of the 
Riemann sphere; 


a b a 
PSL(2,C)> + | | o> a € Méb(C) 
and through the Poincaré extension we identify Mob(C) with Isom*(H®) the group of orientation preserving 
isometries of hyperbolic 3-space. A thorough discussion of these things can be found in Beardon's book [2]. The 
numbers x,y,z defined above at (12.1) are not well defined in PSL(2, C), and so we first identify new parameters. 
Given two matrices A, B € SL(2,C) we define the parameters 


7(A, B) = tr[A, B]—2, (A) = tr?(A) — 4, 6(B) = tr2(B) — 4. 
(12.2) 


Here [A, B] = ABA'B’! is the multiplicative commutator. These parameters depend only on the conjugacy 
class of (A, B) and are well defined in the projective group PSL(2,C). They determine the group (A, B|---) 
uniquely up to conjugacy if 7(A, B) 4 0, [12]. 

Note that 7(A,B) is unchanged by Nielson moves (automorphisms of the free group of rank 2) on the 
generating pair {A, B}, so for instance 


(A, B) = 7(B, A) = 7(A, A™BA") 


and so forth. The parameters are set up so that if (A, B) = I, the trivial group, then (7, 8, B) =0eEC?. 

In this article we are primarily interested in a special family of words, called good words, and the family of 
trace polynomials they generate. These words are defined in Section 12.5 below. This remarkable family W has the 
following properties reminiscent of the Chebyshev polynomials: 


1. [Semigroup structure] W forms a semigroup under the operation 
w3(a, b)*we(a, b) = wi(a, we(a, b)) 


(12.3) 


2. [Polynomials and composition] For each w(a, b) € W there is an associated monic polynomial with integer 
coefficients in two complex variables P,(7y,8). These polynomials have the property that if 
w1(a, b), wo(a, b) € WY, then 


Pru wy (7, B) = Pry, (Pals 6), B) 


(12.4) 


That is the semigroup operation above induces polynomial composition. 


io’) 


. [Commutators and bounded roots] Given a representation of 
T= (a,b | b? — 1) 


into SL(2,C), at+ A, b> B, set y= 7(A, B) and 6B = B(A). Then for w = w(a,b) € I, wis a good 
word and we have 


(a) Commutator independence from the third complex variable 6(B), 


(A, w(A, B)) — Py (7; B) 


(b) Suppose A is not an irrational rotation, equivalently 
BS {-Asin? (rm) :r € R\Q}, 
and denote the zero set of the polynomials by 
Zi = {ze C: P,(z, 8) =0 for some w € I} 
Then # is compact and 
C\ 25 = #5 
where #g is nonempty, unbounded and conformally equivalent to the punctured disk. 


(c) The group (A, B) is discrete and free on generators 
(A, B) = (A)*(B) 
if and only if y(A, B) € Bg. 


Here we will prove the density of the roots of good word polynomials in the exterior of the moduli space of 
discrete and faithful representations of I’, that is 3 (b). The hard part of 3 (c) concerns the structure of the 
boundary, and the only proof we have relies on some very deep results concerning the geometry of discrete groups, 
such as the density and the ending lamination theorems, see [26]. This is because #g can be identified with the 
(moduli) space of discrete and faithful geometrically finite representations of Z*Z2, with the generator of Z = (A) 
and 6(A) = G. The “pants” decomposition of a geometrically finite Riemann surface with fundamental group 
Z*Zy or Zp*Zy shows that #g is topologically a punctured disk. These are obtained from the disk with two cone 
points of order two glued along its boundary to a disk with two holes (or punctures). In the case 8 = 0, & is 
known in the literature as the Riley slice and the boundary 0% is a topological circle [1]. This is expected to 
persist for all other G6 € C as well. The geometrically infinite faithful representations lie in the continuum 
(topological circle ?) 0%. 


The complement C\ Zz consists of nondiscrete groups apart from a countable discrete set in Zg of points 
which are the roots of polynomials corresponding to relators in groups that are discrete but not splitting. There are 
some conjectures about the structure of the polynomials and the words they come from - basically that they are 
associated with Dehn surgeries on two bridge knots and links and associated Hecke groups (obtained by adding an 


unknotting tunnel). The cusp points on the boundary arise from pinching a geodesic (arising from a Farey word) of 
the Riemann surface with fundamental group Z*Z, as in [19], giving a ray in the unbounded region ending on 0&, 
while from the bounded region C\0Z cusps are associated with Dehn surgery limits (via Thurston's Dehn Surgery 
theorem) from the inside. 

Indeed it is the strong connection between these representation spaces of discrete groups, low dimensional 
hyperbolic geometry and topology and the good word polynomials that motivates our consideration of them. The 
geometry of commutators plays an important role in understanding the geometry and topology of discrete groups 
and their associated quotients, hyperbolic 3-manifolds and 3-orbifolds. For instance if A,B € I’ where I is a 
discrete subgroup of SL(2,C), we put 6 = 6(A) and then suppress it writing P,,(y) for P.,(7, 8) to find that 
{Pu(y) : w € HW} is a collection of traces of commutators in I. Further, if w ¢ W, then the semigroup operation 
gives w*w*.--*w € W and 


Ply); Poel) = Py (Poly); tee Putty) = Po) 


gives a sequence of commutator traces from the holomorphic dynamical system given by iteration of the 
polynomial P,,. As perhaps the simplest nontrivial example, with w(a, b) = bab~', we have P,.(y, 8) = y(7 — B). 
If 6 =0, then we see y,y”,...,y”... is a sequence of commutator traces. If 0 < |y| <1, then the sequence 
{y"},59 accumulates on 0. It is not a particularly difficult exercise to show this can’t happen in a discrete group, 
and we therefore obtain the classical Shimitzu-Leutbecher inequality. 


>= 
S— 
— 


\\\ 


FIGURE 12.1 


The Riley slice (shaded region) is symmetric about the origin and its complement meets the real line in the interval |—4, 4] and the imaginary axis in 
[—2, 2]. From David Wright. In [21] it is shown that the exterior - the closure of the complement of the Riley slice - is the Julia set of the good word 
semigroup for groups generated by a parabolic and an elliptic of order two. Also illustrated are the first few thousand roots of the good word 
polynomials. 


Theorem 12.1.2 (Shimitzu-Leutbecher inequality). [f (A,B) C SL(2,C) is discrete and A is parabolic (8 = 0), 
then |y| = |tr[A, B] — 2| > 1. 


Jgrgensen's inequality [18] follows in the same way if |G] <1 for then 0 is an attracting fixed point for the 
iterates of P,, and the disk D(0, 1 — ||) lies in the Fatou set so y ¢ D(0,1— |6|) and so |y| + |6| > 1. We will 


give other examples later. 


In order to fully exploit these polynomials in low dimensional topology and geometry, it is crucial to 
understand more about them and develop a systematic approach to uncovering the inequalities and regions of 
moduli space where their roots lie. For instance, to understand and extend the important Flog 3 theorem of Gabai, 
Meyerhoff and Thurston [8], used to prove the topological rigidity of hyperbolic three manifolds [7], an ad hoc 
approach required rigorous estimates on the computation of 100+ matrix multiplications—these words were called 
killer words as they removed small regions of moduli space using discreteness criteria, such as Jgrgensen's 


inequality or other criteria, such as contradicting a choice of shortest geodesic. An approach based on good words 
is far simpler since estimates are required for the roots of a polynomial equation with integer coefficients of lesser 
degree. Such searches have been used to resolve a number of problems such as: 


1. The unique minimal volume 3-orbifold (co-volume lattice of hyperbolic isometries) identified as the 
arithmetic Coxeter reflection group 3-5-3, extended by the order two symmetry induced from the diagram 
[14, 22]. 

2. Structure of the singular set. Tables 6-10 of [9, 14] give sharp bounds for the distance between components of 
the singular set of a hyperbolic 3-orbifold and the distance between tetraheral, octahedral and icosahedral 
points in a Kleinian group. 


io’) 


. Automorphism groups of 3-manifolds and 3-dimensional Hurwitz groups. Sharp bounds for the order of the 
automorphism group of a hyperbolic 3-manifold group in terms of the volume and analogous to the 
84(g — 1) Theorem of Hurwitz [5] 

4. Margulis constant. The Margulis constant is achieved in a two- or three- generator group, the case of two 

generator groups is completely resolved, [13] and the only remaining case concerns Kleinian groups 

generated by three elements of order two. 


Nn 


. Geodesic length spectrum of 3-folds. Inequalities are used to find bounds on the length of intersecting closed 
geodesics, or non-simple geodesics, which are within a factor of 2 of being sharp. These together with 
estimates on the Margulis constant, yield good bounds for the thick and thin decompositions of hyperbolic 3- 
manifolds. 


a 


Explicit examples of small volume hyperbolic 3-manifolds and 3-orbifolds with various geometric properties 
including a sequence of orbifolds with torsion of order g interpolating between the smallest volume cusped 
orbifold (¢ = 6) and the smallest volume limit orbifold g — oo, hyperbolic 3-manifolds with automorphism 
groups with large orders in relation to volume and in arithmetic progression, and the smallest volume 
hyperbolic manifolds with totally geodesic surfaces in [4]. 


In this paper we uncover a group ¥ of elements of unit norm in a quaternion algebra 2 with associated 
indeterminates, which maps under an “evaluation homomorphism” p : ¥ — PSL(2,C) to a group which includes 
these good words on two generators. Further Y naturally extends to a larger group which gives a corresponding 
extension of the isometry group p(V¥). Roughly, polynomials R,.S,T and Win the indeterminates u and v form a 
“quaternion” (R,S,T,W) € 2, which has norm 1 when 


R? — (u? — 1)S? — (vb? — 1)T? + (wu? — 1)(v? -1)W?2 =1 


(12.5) 


A special case of interest occurs when u or vis +1, in which case this equation reduces to the polynomial Pell 
equation, 


P?(z) — (2? —1)Q?(z) = 1, 
(12.6) 


An obvious similarity between the two equations is that the solution sets have a natural group structure; this is 
what we will exploit to begin to understand the structure of good words. However, there are significant differences. 
For instance while the solutions P(x), Q(x) of the polynomial Pell equation must have integer coefficients, there 
are members of ¥ whose polynomials have coefficients which need not even be rational (see Section 12.8.7 for an 
example). We also note that (12.5) has some solutions with strictly complex coefficients: a simple example is 
R(u,v) = uv, S(u,v) =T (u,v) =1, W(u,v) = 7. However, when we confine ourselves to solutions with 
rational coefficients, some remarkable properties emerge; in particular, it turns out all that such solutions actually 
have half-integer coefficients Section 8. 

In order to study these word polynomials more fully, as well as justify the sorts of results we are seeking, we 
need to develop a few ideas from hyperbolic geometry and, in particular, from the geometry of discrete groups of 
hyperbolic isometries of hyperbolic 3-space. 


12.2 Background in Hyperbolic Geometry 
Let Isom* (HH?) be the group of orientation preserving isometries of HP, hyperbolic 3-space, 


H? = {z = (21,229,273) € R°, 23 > of, with metric ds = Hel 


of constant negative curvature equal to —1. 
We briefly review some well-known facts about the group Isom* (HI); see e.g. [2, 12] or [23] for more details. 


Each f € Isom*(HI*) is the Poincaré extension of a Mébius transformation of the boundary OHI? which we 
identify as G=Cu {oo}, the Riemann sphere. Hence there is a natural isomorphism between Isom‘ (HI?) and 


PSL(2,C). Using the definition at (12.2) we can thus define the trace and £ and y parameters for isometries 
f,g © Isom*(H?), simply by setting tr(f)=tr(A), B(f)=6(A) and +(f,g)=~7(A,B), where 
A,B € PSL(2,C) represent fand g respectively. 


Each non-identity f € Isom* (IH?) has either one or two fixed-points on the boundary C. If there is just one, 
then fis called parabolic; if there are two, then we define the axis of £ ax(f) to be the hyperbolic geodesic line 
joining them. Now f leaves ax(f) invariant, and its action on this geodesic is a translation along by a distance 
7 = T(f) > 0, the translation length of f, together with a rotation through an angle 7 = 7(f), the holonomy of f 
around ax(f). If 7(f) > 0, then 7 € (—7, 7], is taken anticlockwise around ax(f), as determined by the direction 
of the translation of ax(f) performed by f and the right-hand rule; in this case fis called loxodromic. If r(f) = 0, 
that is if f fixes ax(f) pointwise, then fis called elliptic, in which case the distinction between clockwise and 
anticlockwise disappears, and we may assume that 7 > 0, that is 7 € (0, a]. 

When fis elliptic or loxodromic the parameters 7(f) and (f) together determine fup to conjugacy. 

When fis parabolic or the identity, we set r(f) = n(f) = 0. 

The following lemma classifies the isometries in Isom*(H*) up to conjugacy, and identifies, for each isometry, 
the conjugations which leave it unchanged. 


Lemma 12.2.1 A non-identity isometry f € Isom* (HI?) is conjugate to z + 1 if f is parabolic, and otherwise to a 
unique isometry of the form f(z) = rez, wherer = ef) > 1, -1 << O0< an ifr>1,and0 <0<7ifr=1. 


If gfg~! = f, then either g is the identity, g and f have exactly the same fixed points on en or, fis an elliptic of 
order 2, and g is an elliptic of order 2 which interchanges the endpoints of ax(f). 


As previously remarked, both parameters 6(f) and y(f,g) are invariant under conjugacy. Conversely, if 
B(f) #0, then G(f) determines f up to congugacy, and if y(f,g) #0, then G(f), B(g) and >(f,g) together 


determine the group (f, g) up to congugacy [12]. We prove this result in Theorem 12.4.1 below by identifying a 
canonical representation. 
Both the parameters y(f,g) and 8(f) encode geometric information. For instance: 
: 2 r+i 
B(f) = 4sinh (=), 
(12.7) 


and, when fis elliptic or loxodromic, 


(12.8) 


where A = A(ax(f),ax(gfg !)) represents the complex distance between ax(f) and ax(gfg~+) (the imaginary 
part of this distance, which represents the angle between the two axes, is defined modulo 7, so the right-hand side 
of (12.8) is well defined). It is an elementary fact (see e.g. [2] or Theorem 12.4.1 below) that y(f, g) = 0 if and 


only if f and g share a fixed point on the boundary C of HP; indeed for non-parabolic f and g, this follows 


immediately from (12.8), A being 0 when the axes of fand gfg™! either meet at a point of € or coincide. In 
applications we often want to distinguish between these two cases. We develop an algebraic test in Section 12.6. 


Es 
12.3 Matrix Identities 


We collect some matrix identities for later use. Let 


where ad — bc = 1. Then 


d- ke 
NQ? = 
avgt= [he 
Te f° +k me —m?c+mk(d—a) + ” 
[ke d—k-!mc : 


and when m = 0 


M,N] = ke —k*be — ab(k? — 1) | 


cd(k-* —1) ad—k~*be, 


(P, N] = ie oe 


Cc? 1—ac 


In particular we have the useful trace identities, when m = 0 


tr[/M, N] =2—(k—k-!)’bc 


tr[P, N] =24+¢? 


(12.9) 


(12.10) 


(12.11) 


(12.12) 


(12.13) 


(12.14) 


(12.15) 


Common perpendicular Axis(g) 


View down the common perpendicular 


FIGURE 12.2 


Illustrated are two loxodromic elements fand g. The axes are the hyperbolic lines connecting the fixed points in © Each of fand ghas a translation 
length—the distance a point is moved along the axes—denoted r(f) and r(g). The common perpendicular has hyperbolic length 6. When viewed along 
this common perpendicular, the two axes form an angle @. All of these quantities are related by the formula . 


4yV(f.g) gs 2 3 
ACAstay — Sin” (5 + 48) 


(3) 


12.4 Two-Generator Groups 


We now classify up to conjugacy all two-generator subgroup of Isom* (HH), by finding a canonical representative 
for each conjugacy class. Throughout we always use the principle values of square roots. 


Theorem 12.4.1 Every group generated by two non-identity isometries in Isom‘ (H*) is conjugate to a group of 
the form (f, g), where f and g have matrix representatives A and B respectively in PSL(2,C) such that either: 


Case 1. 
BACAHD+HA) 9 
A= 2/B(F) B- k ‘| 
0 BABA LA-Af) |’ ce d 
2/ BCA) 


(12.16) 


(12.17) 
and 
c=-b=,/ Wo 
(12.18) 
when (f,9) # 0, and either 
b=0,c=lorb=1,c=0orb=c=0 
(12.19) 
when (f,g) = 0; or 
Case 2. 
a=[} ‘| a-[ UV 1F.9)| 
Dr VUF,9) VB(g) +4 
(12.20) 
or 
Case 3. 
4 =lo a) 
p —|2lv8G)+4+ V8@)] 4 
0 z1VA(g) +4- VA(9)] 
(12.21) 


where £ = 0 when B(g) 4 0, and can take any complex value when §(g) = 0. 


The three cases are respectively the cases B(f) #0 (f non-parabolic), B(f)=0 and y(f,g) #0 and 


B(f) = 1(f,9) = 0 (fparabolic). 

Proof. We set B = B(f), y = y(f, 9). Suppose first that fis loxodromic or elliptic (6 4 0). By Lemma 12.2.1 
we can conjugate fso that its matrix representative A is diagonal. 

By (12.14) we have that 7 = —bc8, whence 


(12.22) 


We have (a+ d)° —4= B(g), and since B is determined only up to sign, we may thus assume that 
a+d= /B(g) + 4. Together with (12.22), this gives that a and d are either as given by (12.17) or are obtained 


from these by interchanging the values of a and d. Using (12.10), we may then conjugate A and B if necessary, to 
interchange a and d, so that (12.17) holds, and A is still diagonal. 


Let rand s be the diagonal entries of A. We have (r + s) —4= 8, and since A is determined only up to sign, 


we may assume that r+ s = J B(B+4)/ J G. Together with the condition rs = 1, this gives that either A or A~? 


takes the form given by (12.16). Since (ATT, B) and (A, B) are the same group, we may assume that A satisfies 
(12.16). 


Finally, we apply a conjugacy of the type (12.11) (with m = 0) to A and B to adjust the values of b and c, 
leaving A unchanged. If b,c £ 0 (i.e. when y ¥ 0), we can use such a conjugacy to give b are c any values subject 
to (12.22); in particular, we can make (12.18) hold. If exactly one of the values of b and c is nonzero, then we 
conjugate to make it 1. The only other possibility is b = c = 0, so the options given in (12.18) and (12.19) are 
exhaustive. 

Now we suppose that fis parabolic. Using Lemma 12.2.1 we conjugate so that f(z) = z+ 1, so that its matrix 


1 
representative A = F il By (12.15) c? = y, and we may assume, since B is determined only up to sign, that 


c= 4/7. 
Now we have two subcases, determined by whether or not y = 0. If y 4 0, then c £0, and we apply the 
conjugation (12.11) with k = 1 and m = —a/c to A and B, leaving A the same, and changing ato 0, whence 


0-1/7 


B= 
VY +V/8(9) +4 


(12.23) 


Let Bi.) and Bi_) be the matrices obtained by taking the + and - signs respectively in (12.23). We show that 
(A, Buy) and (A, B-)) are conjugate groups in PSL(2,C). A conjugation of the form (12.11) with k =i and 
m =0 takes A to A’! and Bi_) to —By,). Thus (A, Bi-)) is conjugate to (Al, —Bi)) a (A, Bi) in 
PSL(2, C) as required. Thus, without loss of generality, we take the + sign in (12.23). 


If y = 0, then c = 0 so ad = 1. As in previous cases, we may assume that a + d= /B(g) + 4, so that B must 
take the form (12.21), up to an interchange of the diagonal entries. If 6(g) = 0, then these entries are the same, and 
we are done. Otherwise a # d, and we can apply a further conjugation of the form (12.11), with k = 1, so as to get 
both £ = 0 and to leave A unchanged. Now B is diagonal, and interchanging a and d replaces B by B™!. Since this 
operation leaves the group (A, B) unchanged, we may assume that B is given by (12.21). 


Remarks. We can characterize geometrically the four ways of assigning values to b and c given by (12.18) 


and (12.19). In (12.18) y 4 0, so b,c £ 0 and fand g have no common fixed points in C. In this case, as remarked 
in the proof, b and c can be made to take any values whose product is —y/8, and the exact choice is rather 
arbitrary. However, the normalization that we have chosen is quite natural from a geometric viewpoint; it makes 
the fixed points of g mutually reciprocal, and (consequently), when g is non-parabolic, the common perpendicular 
of ax(f) and ax(g) is the geodesic with endpoints +1. When g is parabolic, the fixed point is z = 1. See [22] for 
more details. 


The first two cases of (12.19), when {b, c} = {0, 1} occur when fand g have a single common fixed point in C 
. If fis loxodromic, then this point is repulsive when b = 0, c = 1, and attractive when b = 1, c = 0. If fis elliptic, 


then f rotates H? anticlockwise (resp. clockwise) around ax(f) oriented away from the shared fixed point when 
b=0, c=1 (esp. b=1, c= 0). (When the elliptic fis order two, these two cases are conjugate.) Finally 
b = c= 0 when fand gare both elliptic or loxodromic and have the same axis. 

Note that, when - = 0, although S(f), G(g) and y do not determine the conjugacy class of (f,g), when 
B(f) #0 (and symmetrically when 6(g) #0) then there are only three possibilities. Only when 
B(f) = B(g) = y = 0 (Case 3 of the theorem with 6(g) = 0) do the same parameters give an infinite family of 
non-conjugate groups. 


ee 
12.5 Good Words 


A good word on the letters a and b is a word of the form 


w(a, b) = ba" ba"... bem tarm1pe™ 


where s; = £1, 5; = (S123 and the r; take integer values. 
Thus the powers of b in a good word alternate in sign. By setting r; = 0 (resp. rm_1 = 0), we obtain a good 
word which begins (resp. ends) with a power of a. 


A good word is even if r3 +r2+...Tm_i is even, odd otherwise, balanced if m is even, unbalanced 
otherwise, and regular if s; = 1, irregular otherwise. If r; = 0 for any 1 < 7 < m—1, then w(a, 6) collapses into 
a shorter good word (which has the same balance, parity and regularity as the original word), so we may assume 
that these interior powers are non-zero. 


The following easy observation is quite useful. We leave the proof to the reader. 


Theorem 12.5.1 The regular balanced words in (a,b), say reg, comprise a subgroup of the free group on a and b, 
of which the regular balanced even words form an index-two subgroup. 


Lemma 12.5.1 The group of regular balanced even words, say I-57", on a and b is generated by a”, ba?b-! and 
[b, a] = bab-ta-}. 


Proof. Let w = ba"b~1a™ ... ba"”-1b—! be regular, balanced and even. We use induction on m. We have 
batb— ai = (ba2b-1)*/? (q2)9/? 
(12.24) 
when i and j are both even, and 
Te ee Cee (ba2b-1)"-9? [b, a] (a2) 7)? 
(12.25) 
when i and j are both odd. This deals with the case m = 1 and the induction step when r, and 1, have the same 


parity. 
If r, and 1 have the opposite parity, then m > 1, and we use the same identities together with 


ba™'b-1a™ba"sb-1a"! = (ba"'b-1a"2-1) [b, al" (ba"s+1b-1a"4) 


This completes the proof. 


The next corollary is also immediate. 


Corollary 12.5.1 Suppose that a has order three, a* = 1. Then the group of regular balanced even words on a and 
b is a two generator group generated by a and bab“. 


We recall here the well known identity 


[b,a] = (ba)? (a-!b-1a) (a2)? 


This tells us that the regular balanced words in I’ = (a, b) lie in the group I) generated by squares of elements. 


Corollary 12.5.2 The group of regular balanced even words on a and b lies in the group generated by the four 
squares 


reven < (a, (bab-1)”, (ba)?, (aba~1)’)) 


reg 


The remark following the next result, Theorem 12.5.2, shows that the index between these two groups is infinite. In 
fact, for any representation in SL(2, C), the trace fields are Q(trI\Gy") = Q(8,7) and Q(A, 7, B(ba)), using our 
earlier notation. 


We can now state our first main theorem. 


Theorem 12.5.2 Let w = w(a,b) = ba"b-1a™...ba’"4b-! be a regular balanced even word, then there are 
polynomials rw, Sw, tw, Ww, such that 


2rw, Ww, Ww, 2Ww, Pw — Sw; tw — Ww € Zz, z], 
(12.26) 
rw(0,0) = 1 and 


Sw(x,z)—zWy(z,z) 


Gw(@, 2) = ; 


is also a polynomial, and if f,g € Isom*(H*) are not the identity, have parameters B = B(f), 6’ = B(g) and 
y= (f,g), and if A and B are the matrices from Theorem 12.4.1 which represent (up to conjugacy) f and g 
respectively, then: for f non-parabolic (8 # 0) 


w( _ [rw(B, *) + 228 ab| Btw (8, 7) + Ww(B, 7) Q| 
Lealt (8,4) ~ ww(B.7)Q]_ ru(B,7) ~ 2 


(12.27) 
where Q = /B(B + 4) and a, b, c, d are as in (12.17) and (12.18); and for f parabolic (8 = 0), withy # 0 
Ww ? tw ? 4 WwW ? 2 WwW ? 
w(A, B) = i (0,7) + ytw(0,7) 4gw(0,7) + 2wu(0 ui 
2ywu(0, 7) Tw(0, 7) — ytw(0, 7) 
(12.28) 
and for f parabolic, with y = 0 
1 4 w\Ys _ ( F p : 4) W\Ys 
w(A,B) = [1 49=(0:0) ~ (8+ VB'VB +4) wo(0 ”) 
0 1 
(12.29) 


In particular, the trace tr(w(f,g9)) = 2rw(B,y) € Z[a, z]. 


Remark. A key feature here is that the polynomials r,, s,, ¢,, and w,, and in case (12.28) the whole matrix, are 
independent of 8’ = G(g). In particular this is true of traces of the matrix representations above. 


We prove the above result in Section 12.7. We can also use it to find w(A, B) when w is unbalanced; we do 
this next for non-parabolic f 


Corollary 12.5.3 If w = w(a,b) = ba"b-'a™...b-1a""~b is a regular unbalanced even word, then there are 
polynomials ry, 8y,tw, Ww With half-integer coefficients such that for f non-parabolic 


a(ra(B, 7) af Sw(B, 7)Q) b(tw(B, 7) a Wu(B, 7)Q) 


w(A, B) = c(tw(B, 7) - WB, 7)Q) d(rw(8, 7) a Sw(B, 7)Q) 
(12.30) 


where a, b, c, d are as in (12.17) and (12.19). 


Proof. First note that @ = w*b! is balanced, thus 


ot b(Btg + we 
w(A,B) = (4, B)B = |" one ile 


- weQ) ro — = ed 


B 
a2 + (1— 3) (ts + weQ)) 
clra — “82 + (1— F) (Bts — weQ)] dra — 292 — F(Bta — waQ)] 
te 


oa) ea) bra HIP ate + (get | 
erg + (8 —y)te — (9a +wa)Q] d((ra — yta) — 9aQ) 


using (12.22). We have now found the polynomials for w explicitly in terms of those for w: 


a 
Zz 
=) 
+ 
g 
6 
| 
BR 
— 
ey) 
S 
j 
+ 
€ 
iS 
© 
OES 


Tw =Ta — Ye 

Sw = 9a 

ty =e t+ (B- ta 
Wy = Jer Wa 


We now extend the definition of the polynomials r,, s,, ty and w, to arbitrary good words. 


Definition. Let w be a good word. If wis regular and even, then then the polynomials r,, s,, ¢,, and w,, are as 


defined in Theorem 12.5.2 for balanced w, and Corollary 12.5.3 for unbalanced w. If w is regular and odd, then 
v := w.a is regular and even and we define ry, = Ty, Sy = Sy, ty = ty and Wy = Wy. Finally, if w = w(a, db) is 


irregular, then w’ = w(a, b-') is regular, and we define r,, = ry’ etc. Note that (w. a)’ =w’.a, so that if wis 
irregular and odd, we have ry, = T(w.a)' = Tw'.a and so forth. 


If w=w(a,b) is an irregular even word, then w is regular and even. Since 6(g~') = G(g) and 
a(f, 9°) = y(f,g), Theorem 12.5.2 and Corollary 12.5.3 show that w(f, g) = w’(f, g~') is conjugate to w’(f, g) 
; in particular these have the same trace 2r,,(8, 7) = 2rw (8,7). 


Since for any good word w = w(a, b), the commutator /a,w/ is balanced and even, it follows easily that 
(f, w(f,9)) = Pw(G, 7), for some polynomial p,, € Z[z, z] as first observed in [12]. The next result expresses 
these polynomials in terms of t, and w,,. 


Theorem 12.5.3 Let w= w(a,b) be a good word, 8 = B(f), y=7(f,g) and h=w/(f,g), then there is a 
polynomial py € Z|, z], such that 


V(f,h) = Pw (B,) 
(12.31) 


If w is balanced, then 
Pw(x, Z) = 2(x — z)(xt? (x, z) — (x + 4)w? (2, z)), 
(12.32) 
If w is unbalanced, then 
Pw(z, 2) = 2(t2,(z, z) — x(x + 4)w? (2, z)). 
(12.33) 


Proof for non-parabolic f: We may assume that fand g have matrix representatives A and B, respectively as given 
by (12.16)-(12.19). Suppose first that w is regular and even, and set W = w(A, B). The required identities follow 
from (12.14) and (12.22), together with (12.27) when W is balanced and (12.30) when W is unbalanced. For 
balanced w, py € Za, Z| follows from (12.26). For unbalanced w, using the results and notation of the proof of 
Corollary 12.5.3, we have 


ty —LWy =Pat (x + z)te ~ (9% + we) 
=rgt(e@+z)tg — (sg — zw) — cus 
= (rg — sg) + (e+ z)(tg — wa) € Ze, 2], 


again by (12.26). Consequently, also ¢,, + cw, € Z[x, z], and the same then follows for p,, as given by (12.33). 

If w is odd, the result follows from the previous case, together with the identity [A, W] = [A, WA]. Finally 
suppose that w is irregular, then the commutator /a,w/] is irregular and even, and [a, w)' = [a, w’] is regular. Since, 
as previously noted, tr([a, w]) = tr([a, w]’) = tr([a, w’]), we are reduced to the case where w is regular. 


It is not difficult to prove the above directly when fis parabolic. However, we will instead use a limiting 
argument in Section 12.7 (Corollary 12.7.1). 
12.5.1 Examples of word polynomials 


Before going too much further it is worthwhile giving a few examples of polynomials. These appear in Table 12.1 
below. 


TABLE 12.1 
Some examples of word polynomials. 
Polynomial word 
vy — B) bab 
(B+4)(y— B)y ba2b 
(B-y+1)"7 balee 
(1 — 28 +? — (B - 2)9) baba~'b 
(1+ B(B + 1)(B + 4) — (8 + 4) (28 + 1)y + (6 + 4)7”) babab 
(62 — (y-4)8 - 47 +1)"7 ba*ba%b 
yy — BY(B-y+2)° iaiieseia 
(8? + 7° — 267? + (8 - 1)6y) bababa~*b 
(B+ 4) (6? + 7° — 287? + (8 - 1)6y) baba*ba*b 
(vy — B)(B + 4)(B(7? — 3y — 4) — B2(y + 1) +47? 4+ 474-1) ba~*bababa*bab 


The last polynomial here is quite long, but it has the remarkable property that it has y = 0 as a super-attracting 
fixed point. We will need this fact later. 


LSS 
12.6 Applications 


Inequalities such as Jérgensen's inequality hold quite generally for discrete groups of isometries of negatively 
curved metrics in an appropriate form as it is essentially based on Zassenhaus’ lemma (see eg. [20]). However, it is 
the explicit and sharp nature of these and other inequalities in terms of natural parameters which yield good 
information in three dimensional hyperbolic geometry. 


If (f, g) is discrete and non-elementary, which in this setting the latter means that (f, g) is not virtually abelian, 
then we may recover Jgrgensen's inequality, [18] as follows. Recall we sketched a proof for this earlier at Theorem 
T21.2. 


Theorem 12.6.1 Let (f, 9) be a discrete nonelementary subgroup of SL(2,C). Then 


IB(f)| + lyf, 9)| = 1. 
(12.34) 


If wis a good word and if h = w(f,g), then we have 7(f, h) = py(7, 8), where p,, is the polynomial of Theorem 
12.5.3 and so we deduce from Jgrgensen's inequality applied to the group (f, 2), that 


IGF) + l(f,2)| = 18] + lpw(8,7)| 2 1 


unless +(f, h) = 0, in which case (f, h) is elementary. 
We would like to understand how this exception happens more generally. 


If |B(f)| + ly(f, h)| < 1 and y(f, h) # 0, then (f, g) is non-elementary since fis either loxodromic or elliptic 
of order at least 7. Then Jgrgensen's inequality implies that (f, g) cannot be discrete. On the other hand, this group 
may be elementary if +(f,h) = 0. However, in this case, we know that the fixed point sets of f and of A on c 
intersect, and they may coincide; for non-parabolic fand h, coincidence means that they have the same axis. We 
can use polynomials to determine when this happens. 


Proposition 12.6.1 Let (f,g) be discrete and non-elementary, and h=w(f,g) for a good word w. Set 


B= B(f) 4 —4, and y=~+(f,g) # 8,0. Then the fixed point sets of f and h on © coincide if and only if 
tn (8, 7) = Ww(B, 7) = 0 
Proof. We suppose first that w is regular and even. We may suppose that fand g are normalized so that their matrix 
representatives are as given in Theorem 12.4.1 (specifically, by (12.16)-(12.19) for 6 #0, and by (12.21) for 
8 = 0), and consequently that the matrix representative for his given by Theorem 12.5.2. 

Suppose first that 6 4 0, so that fis loxodromic or elliptic with fixed points 0 and ~ on the boundary. Since 
abcd = >(y — B)/B” # 0, (12.27) and (12.30) show that A shares these fixed points if and only if 


Btw(B, 7) = Ww(B, 7)Q _ 0, 


when w is balanced, and 


tw(B, 7) xo Ww/(B, 7)Q = 0, 


when it is not. Since also 8 4 0 and Q £ 0, this is equivalent to ty(8,y) = ww(G, 7) = 0 in both cases. 

Now suppose first that 8 = 0, so that f(z) = z+ 1 with fixed point ~. If © is also the only fixed point of A, 
then A must also be parabolic, and by (12.28), we must have r,,(0,y) = 1 and w,,(0, y) = 0, in order to get the 
right trace and the right fixed point respectively. The determinant condition then gives t,,(0,y) = 0. The converse 
is clear. 


If wis regular and odd, then the fixed point sets of fand A on C coincide if and only if the same is true of the 
fixed point sets of fand hf Since w.a is even this occurs exactly when t,,., = Wy. = 0, by the previous case, and 
since ty = ty. and Wy = Wy, we are done. If wis irregular then h = w'(f,g +), so that, since 8(g-1) = B(g), 


9(f, 9-1) = y(f,g) and w is regular, the previous cases give that the fixed point sets of fand h on € coincide if 
and only if ty (8,7) = ww (8, y) = 0, and we are done, since by definition, t,, = t, and wy = Wy. 


The above, together with (12.32) and (12.33), gives the following corollary which is useful. It allows us to 
obtain a contradiction to discreteness unless we can identify a multiple root, which is easy to do computationally, 
using the vanishing of a discriminant. 


Corollary 12.6.1. Let £ g and h be as above and let B 4 —4, B 4 y and y ¥ 0. If f and h have the same fixed 
points in C, then z = ¥ is a multiple root of py(B, Z). 


Remark. The converse of this result is false. For example, when fis parabolic, fand A cannot have the same fixed 
points on the boundary unless h is also parabolic, but inspection of (12.32) and (12.33) shows that p,,(0, z) has no 
simple roots apart from z = 0. 


| 
12.7 Quaternion Algebras 


In order to prove Theorem 12.5.2, we switch from matrices into the slightly more abstract setting of quaternion 
algebras. The objects we will be dealing with here are essentially the same as matrices of the form (12.27), but 
with numbers replaced by indeterminates (we will let x and z correspond to f and y respectively), and with square 
roots defined abstractly. We first recall some basic facts; see [24] or [30] for more details. 


For each field F of characteristic 4 2 and non-zero a,b € F, the quaternion algebra 
a,b 
wale) 
(12.35) 


is defined to be the associative algebra over the field F with multiplicative identity 1 and basis {1, i,j,k}, with 
multiplication determined by i? = a, j? = b, ij = —ji = k, whence also k? = —ab, jk = —bi and ki = —aj. The 
generic member of is thus x + yi + zj + wk, where a, y, z, w € F; we may abbreviate this to (2, y, z, w). 


& is also isomorphic to the algebra of matrices of the form 


z+ y€1 (z + w61)&2 
(z—wki)E2 «— yr 


(12.36) 
over the extension field F(£1, £2), where £? = a and £3 = b. 
If n = (x, y, z,w) € &, then the conjugate of nis given by 
n = (#,—y, —z, —w), 
and the norm of n by 


2 _ ay? — b(z? — aw?) € F. 


N(n) —nn= xr? — ay” — bz? + abw? =2 
Note that the norm becomes the determinant under the mapping (12.36). 


For reference, multiplication in the quaternion algebra (12.35) is given explicitly by 


(%, Y, 2, w) (x', y', Z, w') 
= (xa' + ayy’ + bzz' — abww’', ry’ + yx' + b(wz' — zw’), 
xz’ + ze’ +a(yw' — wy’), ew’ + wa! + yz’ — zy’). 


(12.37) 


If the field F is the field of fractions of an integrally closed integral domain R, we define an R-Jattice in & to 
be a finitely generated R-module L in 2; L is an ideal if FL = &. An element a € & is an integer (over R) if 
Ria] is an R-lattice. An order in & is an ideal which is also ring with 1. By contrast with the commutative case, 
the set of all integers in .& is not generally a ring. 


We will be particularly concerned with elements of norm 1. Note that these are units, and in any order which is 
closed under conjugation, they form a multiplicative group. 


We consider the following quaternion algebra over the field of rational functions in two indeterminates. 


_ [{ («+4)/2z,2(z-z) 
Qoi= ( R(z,z) ) 


Theorem 12.7.1. The set ¥% of elements of norm 1 of Qo of the form 


[r(a, z) + s(a, z)i+ t(2, z)j + w(2, z)k], 


where 
2r,2s,2t,2w € Zz, z], 
(12.38) 
r(0,0) =1 

(12.39) 

and 

s(x, z) = zw(z, z) moda, 

(12.40) 


is a group. This group is not trivial as, for instance, 


1 1 1 
{e424 al), (e424 (¢ 2z)i ak), (e+2 zi-j wher 


Proof. Let u=(r,s,t,w) = (1/2)(ri,si,ti1,wi) € %. The fact that this has norm 1_ gives 
u-! = u = (r, —s, —t, —w), so clearly % is closed under inversion. We need only show that it is also closed 
under multiplication. We have 


ro — (21) 6? — xz—3)t2 + (2 )e(z—2)wi =4 


(12.41) 


Reducing modulo 2 gives 


z—a)(w;—t)? mod2, 


whence r; — 8; = w; — t; = 0 mod 2, and so 
r1(x,z) = s1(z,z) mod 2, 
(12.42) 
ti (x, z) = wi(2, z) mod 2. 
(12.43) 


Let uz = (1/2) (ro, 52, t2, w2), ug = (1/2) (rz, 53, t3,w3) € U, then using (12.37), ugug = (1/2)(r,s,t,w), 
where 
2r = rors + (2*)5053 + 2(z— a)tats — 2(z — x) (24) wows 
=ror3 + 2(z—ax)tets + (x@ + 4)zwows 
a (x ae 4) ( $2(s3—zw3) ae se] 
2s =12983 + Sar3 + 2(z— £)(wats — tows) 
2t =rot3+ ter3 + (24) (sows -_ W283) 


2w =17rew3 + wer3 + Sot3 — tos83 


(12.44) 


The congruence (12.40) applied to uy and u3 shows that each of these is a polynomial. Moreover, since (12.40) also 
gives s2(0,0) = 0, setting « = z = 0 in (12.44) then gives (12.39). Next 


2(s — zw) = (r2 + 2t2)(s3 — zw3) + (r3 — 2t3)(s2 — zwe) + v2(tow3 — wots), 
so (12.40) holds for ugus3. It remains to show that the polynomials r, s, tand whave integer coefficients. We have 


2r =ror3 + (4*)s283 + 2(z— a)tets — 2(z — x) (+*) wows 
= ror3 + $983 + 2(z— £)(tat3 — wows) mod 2 
= 0 mod2, 


(12.45) 


using (12.42) and (12.43), whence r € Z/z, z]. Similar, and easier, arguments give the same conclusion for s, t and 
Ww. 


Remark. It is not difficult to show that r(0,0) = +1 and s(za, z) = +zw(a,z) modz follow from (12.41), so 
that (12.39) and (12.40) are just normalizing choices of sign. 


For each fixed 8, 8’, y € C, with 6 0, if we let Q = ./8(G +4) and D, and D, be any fixed numbers such 
that Dj D2 = 7(y — B)/?, then the evaluation map 


6,9'q([r(@, 2) + s(a, 2)i + t(@, z)j + w(a, 2)k)) 
_ |r, 4) + “ge D,(6t(B, 7) + w(B, 1Q)| 
~ [D2(6t(8,1) — w(8, YQ) (8,7) — “72 
(12.46) 


is an algebra homomorphism from Qo to M2(C), the algebra of 2 x 2 matrices over C. 
For 6 = 0, y 4 0, we set 


0,6'y([r(x, z) + s(x, z)it t(a, z)j + w(z, z)k]) 
7(0,7) + 74(0, 7) 49(0, 7) + 2w(0, 7) 
2y~w(0, 7) r(0,y) — 7#(0, 7) 


where g (a polynomial by (12.40)) is given by 


s(x,z)—zw(z,z) 


g(x, z) 3S ge 


and 
¢$0,60([r(#, z) + s(a, z)i + t(a, z)j + w(a, z)k]) 


2 Ks 0) 4g(0,0) — (3 " VBV/B'+4)w(0, "| 


0 r(0,0) 


(12.47) 


(12.48) 


(12.49) 


The next theorem shows that the maps ¢o,g',, arise as limits of maps $4", (after conjugating @ 4,7 in such a 
way as to make its fixed points approach a common limit as 8 —> 0). It follows that the maps ¢g g',, are all algebra 
homomorphisms (this is also not difficult to show directly). In particular, each ¢g ,, restricted to Vp is a group 


homomorphism to SL(2, C), and thence by projection to PSL(2, C). 


Theorem 12.7.2 Suppose that B £ 0, y 4 0, k? = 1/4/8, and that 


Q =Vb8+4), Q= VBVB+4, 
gt EE) m= Re 


D, =-4,/3(vVF+i+ (2), 
D, = 4/3(VPrt- (BE) 


(1.e. D, = ab and D2 = cd, where a, b, c, d are given by (12.17) and (12.18)), 


mnie = 


! 2h a, 7 2VIV1+88 TA), 
Di = (24)( pray we 
: Vi Ta wai TRB) 
Dy; = (34) VB +4 VB ), 
| vee | ifQ’=Q 
0 /D}/D2 
C a. 
0 is/D'/D2 ete 
i/Di/D; 0 


Then for y # 0, x € J, 


limM$p,9,y(x)M~" = $0,6'(x) 
(12.50) 
and 
lim M1 $0,39(*)My* = $0,61,0(%) 
(12.51) 
Proof. Since DED; = Dj,Dzp, the diagonal entries of C are the same, in the case Q’ = Q, and the off-diagonal 


entries of C divided by i are mutually reciprocal, otherwise. Thus we can apply (12.11) and (12.10) respectively to 
obtain 


Cbp,4(x)C = rte, eee saa Di (Bt(8, 7) + w(8,7)Q')| 
| |4(6t(8,) — w(6,1)Q") (6,7) - 22" | 
Thus 
Mbson(ayM— = [1 20), 


(12.52) 


where, using (12.11), and writing r(8, 7) = r ete. 


_ sQ’ -2| 1 Vpi+4 Wal 
as aaa ia aa 277 tee 


-( ew AVERTED |g wQ") 


_ sQ! vi veya 4, 
a i [t+ 2/7 


(Va +4- awe + O( (/B)) (Vat = 2w + 0(8)) 


Thus 


yon = tnles FS (t4 BEB) (VTA) ea) 


=lim|r+ 25 + ya(1+ EEE) (ty7 + w VB +4 wr) 


Bool VB 2/7 
=r [ 2s (¢ ~ r+ 4— 7 4) - 21] 
lime + 7 + V7 Vi t+ wr/ 8 +4—wi/ hi + 7: 
_}; 2(s—yw) 7 
= lim|r + 222 + ty] = (0,9) + 24(0,7), 


using (12.40) at the last step. Since conjugation preserves traces, we then have lima = r(0,7) — 7t(0,7). 
> 


Qa21 =k (25) (vara ~ SAVEETED ) (57 — wQ 


= (£) ( JB +4 NVTSPTED ) (5 wQ') 


SO 


liman = (27) lim ( =) ( Qw B) = 2yw(0,7) = 28(0,7), 
(12.53) 


again using (12.40). 
Finally we show that lima = 2w(0, 7) + 49(0, 7). Since Det(¢g,4/,,) = 1, and this determinant is preserved 
> 


under conjugation and limits, it suffices to show that 


(r(0, y) + yt(0, y))(r(0, 7) — 7#(0, y)) — 28(0, 7)(2w(0, 7) + 49(0, 7)) = 1. 


This is readily verified by letting 6 — 0 in (12.41), keeping in mind the definition of g, (12.48). This completes the 
proof of (12.50). A similar, but much easier, calculation gives (12.51). Forming the conjugate 


OT dew won ven) Ilo 2”) 


using (12.11), and letting y — 0, gives the matrix at (12.49). 


12.7.1 Proof of Theorem 12.5.2 


Let 
wi = $(a + 2,2,0,0), 
wa = 5(#+2,2—2z,0,—2) 
ws = 4(z+2,-z,-1,-1). 
(12.54) 
Each w; € %. Let A, B be as in Theorem 12.4.1, Q = ./6(8 + 4). In the first case, 8 4 0, we calculate 
Bat k +2+Q 0 | 
~~ 710 B+2-Q 
(12.55) 
Besa —2abQ 
* |2cdQ B+2—(B—27)Q/6 
(12.56) 
_1[y¥+2—7Q/B —ab(6 + Q) 
[B.A] =F] on ; 
cd(8-Q) y+2+7Q/B, 
(12.57) 


In the second case, 8 = 0, y 4 0, we have 


ara [i 
~ fo 1p 


(12.58) 


—2y 1 
(12.59) 
Bal-|' >) | 
=p ed 
(12.60) 
Finally, if 8 = y = 0, then A? is still given by (12.58), and 
pap — |} 2+A(9) + VB(9)VA(9) + 4 
0 1 
(12.61) 
and 
B,a)— | (6) + VEOH) +4) fl 
0 1 
(12.62) 


In this last case the matrices do depend on §(g), but are independent of the parameter ¢. 
Thus, in all cases, a straightforward calculation using the evaluation map at (12.46) gives 


A? = b6¢,(wi), BA?B™ = $8,9,7(w2), [B, A] = $5,6y(ws), 


where = B(f), 8’ = B(g) and y = y(f,g), and where (in the case B #0), we set D; = ab and D2 = cd with 
a,b,c,d given by (12.17) and (12.19). Since, by Lemma 12.5.2, these words generate all regular balanced even 
words in A and B, it follows immediately that every such word is ¢g 4 y(w) for some w € %. Theorem 12.5.2 
then follows, using Theorem 12.7.1, (12.42) and (12.43). 


Corollary 12.7.1 Let A(8) and B(8,6',y) be the matrix representatives of f and g respectively given by 
Theorem 12.4.1, where now we have made the dependency on parameters 3 = B(f), B’ = B(g) and y = y(f,g) 
explicit. Let w(a, b) be a regular balanced even word. Let 8 # 0, and k, M and M, as in Theorem 12.7.2, then for 


7 #0, 


limM (w(A(8), B(8, 6, 7)))M~* = w(A(0), B(0, 6’, 7)) 


B-0 
(12.63) 
and 
lim M, (w(A(0), B(0, 6’, 7)))M,* = w(A(0), B(O, 8’, 0)) 
(12.64) 


Since polynomials and the trace function are continuous, and trace is preserved under conjugation, Theorem 12.5.3 
for B = 0 follows from the case 6 ¥ 0, by letting 8 — 0. 


12.7.2 A Change of Variable 


We now introduce two new parameters which can be used to describe 2-generator groups (up to conjugacy), and 
which, when 6 ¥ 0, can be used interchangeably with # and y and will simplify many formulas in what follows. 
For f,g € Isom* (H?) we define 


A= ACA) = (te2(f) — 2)/2 = cosh(r + én), 
(12.65) 
so A = (8(f) + 2)/2. When fis elliptic or loxodromic, 


tr?(f)—2tr[f, 
u=nu(f,9) = at a 


(12.66) 


In terms of our earlier parameters 


= 2(f,9) 
H=1— aH 


Rewriting (12.7) and (12.8) in terms of A(f) and u(f, g) gives 


A(f) = cosh(r(f) + in(f)), 


and 


uf; 9) =1- (A(g) oa 1)sinh” (A), 


where we recall A—the complex distance between axes—is defined above at (12.8). An important special case is 
captured by the next lemma. 


Lemma 12.7.1 If g is order 2, \(g) = —1, and we obtain the particularly simple form: 


u(f,9) = cosh(2A). 


Unwinding these parameters gives 


B(f) = 2(A(f) — 1), and y(f, 9) = —(A(F) — 1)(H(F, 9) — 1) 
(12.67) 
If X= A(f) # 1 then it determines f up to congugacy, and if further, u = u(f,g) #1, then A and wu together 


determine the group (f,g). When A # 1, we can rewrite the matrices A and B at (12.16) in terms of the new 
parameters as 


V\=1+(A-1) 


= 10 
aes 2(\-1) == b i 
VN—1-(A-1) 
2(\—1) 


(12.68) 


where now, writing A(g) = ', 


a= + (VN 414+ Y¥ =), d= 4-(VN41- Y=), 


(12.69) 


and band c are given by (12.19) when yu = 1, and otherwise 


en b= 
(12.70) 
for up #1. 
Given a regular balanced even word w, we can now rewrite the matrix w(A, B) at (12.27) as 
Ry (A,B) + Sw(A, pV? — 1 2ab(T,, (A, #) + Wa (A, )V 2 — 1) 
Qcd(Tw(A, 1) — Ww (A, w)VA2 — 1 Ra (A, 1) — Sw(A, w) VA? — 1 
(12.71) 


where a, b, c, d are given by (12.69) and (12.70), and, setting « = 2(u—1), z= —(u—1)(v—1), the 
polynomials R,,, Sy, 7, Wy are given by 


R,(u,v) =1ry(2,2), Wu(u,v) = we (2, 2) 
(12.72) 

S,(u,v) = 28u(2,2)/e, Ty(u,v) = ety (x, z)/2. 
(12.73) 


The congruence (12.40) ensures that S,, is a polynomial. For arbitrary balanced words we take the above as 
definitions of R,, etc. Recall that, in this case, there is a regular even balanced word v such that r,, = ry, Sw = Sy 


etc., so that it remains true that S|, is a polynomial in this case. 


Each of the polynomials 2R,,, 25, 2T,,, and 2W, has integer coefficients, and corresponding to these new 
parameters, we define the quaternion algebra 2 by 


(12.74) 


Here, the indeterminates u and v correspond to 4 and yu respectively. It is straightforward to show that the map 
p: 2 — Bgiven by 


p(r, s,t,w) = (r, ~27, (u— 1)t, w) 
(12.75) 
is an isomorphism. On the right-hand side, x and z are converted into terms of u and v by the formulae 
e=2(u—1), z=—(u—1)(v-1), (u= , v=1- *), 
(12.76) 


these conversions being just the same as those relating £ and y to 4 and wu. The proof uses the observations that 


u2—1= = (24) andy? =1= 4 2(2= 2). 


It is then easy to see that the inverse map is given by 


p (R,S,T,W) = (R, £5, =T, W) 
(12.77) 
Where we now use the second pair of equations in (12.76) to convert the right-hand side back into terms of x and z. 


We can now characterize the image under p of the group % defined in Theorem 12.7.1. This result is a direct 
consequence of (12.75), (12.76), (12.77) and the definition of %. 


Theorem 12.7.3 ¥Y := p(%) comprises the elements 
(R,S,T,W) = (R(u, v), S(u, v), T(u, v), W(u, v)) 


of 2 for which R(1, 1) = 1, and each of 2R, 2(u — 1), 2(u—1)'T, 2W and S + (v —1)W is a polynomial of 
the form 


S> Onm(u — 1)™(v — 1)” 
such that, for each term,m > n and Gn,m Is an integer multiple of 2” ~” (in particular, each ay m 1s an integer). 
Remark. The condition on S(u,v)+(v—1)W(u,v) is equivalent to (12.40): we have 
g(x, z) := (s(x, z) — zw(a, z))/e# = (S(u, v) + (v—1)W(u,v))/2. Note also that this condition insures that 
S(u, v) is a polynomial. 
If x = (r(z, z), s(a, z), t(, z), w(x, z)) € %, and if 
p(x) = (R(u, v), S(u, v), T(u,v), W(u, v)) € Y, 


then by the definition of p, the matrix at (12.71) is $g,7(x) (8 #0). Accordingly we define, for each 
A # 1,.',u € C, the algebra homomorphisms ¢),),, : 2 —- M2(C) by 


Dre (R, S, T, W) 


_ [ROA H) + SQA,p)V 1 2ab(T(A, 2) + W(A, w)Vr? — 1) 
Qcd(T(A, w) + WA, w)VA?2 —1) ROA, pw) — S(A, pw) Vd? — 1 


(12.78) 
where a, b, c, dare given by (12.69) and (12.70). We thus have 
Proposition 12.7.1 For 8 4 0,6’, yE C, x € %, 
Pr,» u(Ps)) = Gs,67(X), 
where X = 1+ 8/2,  =1+ B'/2 andp =1—- 24/8. 


In some respects 2 and wu are better parameters to use than f and y: they have a simpler geometrical 
interpretation, the matrix representations and quaternion algebras are simpler and neater, and there is an obvious 
symmetry between A and wu, corresponding to the symmetry between two loxodromics with perpendicular axes 
(SubSection 12.8.2 below). They also have a major drawback: w is undefined when fis parabolic, so to deal with 
this case we still need f and y. 


12.8 Elements of Unit Norm in 2 


We have found a group ¥ of elements of norm 1 in 2, which maps under each evaluation homomorphism %),y/, 
to a group which includes the regular balanced even words in two generators f and g, where f is elliptic or 
loxodromic. In this section will show how ¥ naturally extends to a larger group which we will denote Y, and this 
gives a corresponding extension of the isometry group ~),y,,(¥%). We look further at this group in Section 12.8.2. 
To begin with we consider properties of elements of norm 1 in general. 

The requirement that (R, S,T,W) € 2has norm 1 is given explicitly by 


R? — (uw? — 1)8? — (v? —1)T? + (Ww -1)(v? —1)W? = 1 
(12.79) 


We will confine our attention to the solutions of (12.79) for which R,5S, 7; W are all polynomials, with the additional 
normalizing condition that R(1,1) = 1. These solutions clearly form a group, which we denote by %. 
We define the degree of u = (R,S,T,W) € % by 


deg(u) = max{deg(R), deg(S) + 1, deg(T) + 1, deg(W) + 2}. 


It is easy to check that deg(uv) < deg(u) + deg(v). For any fixed degree it is possible in principle to evaluate all 
members of % of any fixed degree d, by equating coefficients in (12.79), and we have done this for d < 4. Table 
12.2 lists all members of % of degree at most 2 (up to sign changes of the components 5S, Tand W). In this case, 
the polynomials, like those of ¥, all have integer or half-integer coefficients. Our main result in this section is that 
all members of 2 with components in Q/w, v] and norm in Z/w, v] have this property, and that they form an order 
(with reference to the underlying ring Z/u, v]). We define 


6G = {u=(R,S,T,W) € 2|R,S,T,W © Qlu,v], N(u) € Z[u, v]} 


TABLE 12.2 
R,S,T,W) € % of degree at most 2. 
R Ss i i Ww 
1 0 0 0 
ne ee Cs i sn 
Vv 0 1 0 
u v 0 1 
Vv 0 u 1 
uv 1 u 0 
uv v 1 0 
uv v u 1 
2u? —1 2u 0 0 
2v? 1 0 2v 0 
(1+u+v—uv)/2 (v—1)/2 (u—1)/2 1/2 
(1+u—v-+uv)/2 (v+1)/2 (u—1)/2 1/2 
(1—u+vu-+uv)/2 (v—1)/2 (u+1)/2 1/2 
(-1l+u+v-+uv)/2 (v+1)/2 (w+ 1)/2 1/2 


Theorem 12.8.1. 


1. © is the set of quaternions of the form (R,S,T,W) + +P(u,v)((ut+1)(v+1),v+1,u+1,1), 
where R, S,T,W, P € Z[u, v}. 
2. @ is the unique maximal order of 2, which contains i and j. 


It follows (since @ is clearly closed under conjugation) that the elements of @ of norm 1 form a group, which 
we denote Y. 

The appearance of half-integer coefficients in @ is reminiscent of the Hurwitz order # in H, the quaternions 
of Hamilton, defined as # = {$(n1 + ni + n3j + nak) ¢ H | N1,N2,N3,N4 € Zn, = Ng =Nn3 =n4mod 2} 
. Theorem 12.8.1 has no analog for #; there are plenty of quaternions with integer norm and components which 
are rational but not half-integers, for example (3/5, 4/5, 0,0). However, both the characterizations of @ given in 
the Theorem have their counterparts for #@. It is easy to see that # comprises the quaternions of the form 


u+ +(1, 1,1,1), where u € Z*, and corresponding to Theorem 12.8.1 (2), we have the following classical result 
(see e.g. [30]). 


Theorem 12.8.2 # is the unique maximal order of H, which contains i and j. 


We will take an axiomatic approach which covers both of these theorems. 
We may also characterize # as the set of quaternions with integer norm and half-integer components; this 
amounts to the simple observation that, for integers a,b,c,d 


a? +b? +c +d? =0mo0d4 > a=b=c=dmod2. 
(12.80) 
The integers also satisfy the similar property 
a? +b? +c?+d? =0m0d8 > a=b=c=d= 0mod2. 
(12.81) 


As a simple application of this we observe that a quaternion u with rational components and integer norm has no 
component with denominator divisible by 4. For if this occurred we would have, clearing denominators, a 
quaternion with integer components not all even, and norm divisible by 16, contrary to (12.81). 

We say that a commutative ring which satisfies (12.80) or (12.81) has the four squares property and the strong 
four squares property, respectively. To justify this terminology, we show that (12.81) = (12.80). Suppose that 
(12.81) holds, and that a? + b? + c? + d* = 0mod4, then 
(a —b)? + (a+b)? + (c—d)*? + (c+ d)? = 2(a? + b? + c? + d?) = 0mod 8, and so applying (12.81), we get 
a =bmod 2 and c=dmod 2. The same argument with b and c interchanged gives a =cmod 2, and so 

=b=c=dmod 2, proving (12.80). The converse fails in general (consider, for example, R = Z,4), but holds 
when R is an integral domain. In this case, if R has characteristic 2, then 1? + 1?+0? + 0? = 0, so that (12.80) 
fails, and the implication is vacuous. Otherwise suppose that (12.80) holds, and a? + b? + c? + d? = 0mod8, 
then by (12.80) a=b=c=dmod2, and we have 


( ay? + ( a gee ( esa? + (44 )? = $(a? +b? + c? +d?) =Omod4. Applying (12.80) again gives 
(*)= = (2) = = (S*)= = (4) = 0mod 2, whence it easily follows that a = b =c=d=0mod2. 

The main step in our proofs is to show that, if R has the (strong) four squares property, then the polynomial 
rings R[x1, 2,...,%n] also have this property, together with some generalizations thereof. 


Lemma 12.8.1 Suppose R is a commutative ring and suppose 9; (1 <i< 4) are fixed polynomials in 
R|x1, 22,...,£n] such that 


1. the constant term in each 9; is 1, 


P2 


2. for each non-constant monomial rx}? x” ... x7" in each 9; at least one of the powers p; is odd, 


3.91 = 2 = 93 = ys = 0 mod 2, 


Then for allk > 1, if 


Foralla € R,a? =0mod2 > a= 0mod2,whenk = 1 


(12.82) 
Rhasthe foursquaresproperty, whenk = 2 
(12.83) 
Rhasthestrongfoursquaresproperty, whenk > 3 
(12.84) 
and, for pi, p2, p3,p4 © Rixi,@2,...,2n| 
YiPt + Yop5 + Y3P3 + papy = Omod 2%, 
then 


‘Pp. = po = p3 = pamod 24/2 whenkis even 


“P1 = po = p3 = pa = Omod 2*-Y/? and 
pit p2+p3t+psa= 0 mod 2(*+1)/2 when kis odd 


Proof. We first note that (12.84)=>(12.83)=>(12.82). We have already seen the first of these implications; for the 
second note that if (12.82) fails then there is r € R with r? = Omod2, r = 0mod 2, in which case (12.83) fails 
with a = b=r,c =d =O. Thus the hypotheses of the lemma for any k imply those for all smaller k. 

For convenience, we suppose that n = 2 (the proof for n > 2 is an obvious generalization of this). Throughout 


this proof we order R? lexicographically, that is (a,b) < (c,d) when either a < cora=candb <d. 
Suppose that polynomials y;(x, y) = SoG (m,n)xc™y” € Rix, y] are as in the statement of the lemma, and 


that p;(z, y) = So ai(m, n)z™y” € Rix, y]. 
We use induction on k. Let k € N, and suppose that the theorem holds for smaller values. The hypotheses are 


yi(z,y)pi(x) + pa(z, y)p3(z) + v3(x, y)p3(x) + paz, y)pi(x) = Omod 2°. 
(12.85) 


together with the conditions (12.82), (12.83) and (12.84) on R according ask = 1,k = 2 or k > 3. First suppose 
that k = k; > 3, then by the case k = 3 each p; = 0mod 2, and by the induction hypothesis we apply the case 
k = k, — 2 to the p;/2 € R[z, y| to get the required result. We suppose then that k < 3. 

For k = 1, 2,3 respectively, the required result can be stated in terms of coefficients as, for alln,m € Z, 


-ai(n,m) + ag(n,m) + a3(n,m) + a4(n,m) = 0mod2 


(12.86) 
a1(n,m) = ag(n,m) = a3(n,m) = a4(n, m) mod 2 
ai(n,m) = ag(n,m) = a3(n,m) = ag(n, m) = 0Omod 2 and 
ai(n,m) + a2(n,m) + a3(n,m) + ag(n,m) = 0Omod 4 

(12.88) 


c(m,n) = (c1(m,n),co(m, n), c3(m, n), c4(m, n)) 
a(m,n) = (ai(m,n), a2(m,n), a3(m, 7), a4(m,n)) 
s(m,n) = (si(m,n), s2(m, n), s3(m,n), 4(m, n)). 


We have, for p and g even, 


Si(P,9) = 2D n<(p/2,9/2) ) ai(s, t)ai(p — s,q — t) + a3(p/2, q/2) 


(12.89) 
and, for p or g odd 
si(P,q) = = 207 s,t)<(p/2,q/2) @ ai(s,t)ai(p — s,q—t), 
(12.90) 
whereupon summing gives us 
4 4 
Yosilp,g=2 SY) als,t)-a(p—s,q—t)+ > ai (p/2,4/2), 
< (s,t)<(p/2,4/2) i=1 
(12.91) 


with the last sum only present when p and q are even. 
Equating the coefficient of x?” y?" in the left side of (12.85) to 0 mod 2* gives, using the second hypotheses 
on the @,, 


4 
De php ougigieam 3 (Py a) )+ 8;(2n, 2m) = 0 mod 2°. 
p or qis odd i=1 


(12.92) 
Py (12.90), each term in the first sum is even, whence using (12.91) with p=2n, q=2m, 
oh n,m) = 0mod2. Since S/ a?( n,m) = (Sal n,m )) mod 2, (12.82) gives the congruence (12.86). 
i=1 


For k = 1, this completes the proof. 


We now prove (12.88) for k = 2 and (12.88) for k = 3 by induction on (n,m). Suppose the result holds for all 
(s,t) < (n,m). We first show, for p < 2n, q < 2m with p or q odd, that 


si(p,q) =Omod 2* 1, (i = 1,2,3,4) 


(12.93) 
4 4 
S> 8,(2n,2m) = S> a?(n,m)mod2*, and 
i=l i=l 
(12.94) 
4 
S> 8;(p,q) =Omod2*, 
i=1 
(12.95) 


The first of these follows from (12.90) and the induction hypothesis, and the other two from (12.91) since, for 
(s,t) < (p/2,q/2) < (n,m), and k = 2 


a(s,t)-a(p—s,q—t) = ar(srt) Dale — s,q—t) 


= 0mod2, 


using the induction hypothesis at the first congruence, and (12.86) at the second. 
For k = 3, by the induction hypothesis, each a;(s, t) is even, for (s,t) < (p/2,q/2) < (n,m), and 


4 

ai(s,t 

+a(s,t) -a(p s,q—t) =ai(p—s,q > 0) 
i=1 


= 0mod 2 by (12.88). 


Here we use the lemma for k = 2, which gives (12.88), at the first step and the induction hypothesis at the second. 
Recalling that ci(p’, gq’) = co(p’, ¢') = c3(p’, g') = ca(p’, g’) mod 2, by the third hypothesis on the g, and 
using (12.93), the summand in the first sum of (12.92) is 
4 
s(p,q)-e(p',q') = >_ si(p, q)ci(p’,q’) 
i=1 


a 


i=1 
= 0mod 2* (by (12.95)) 


This, together with (12.92) and (12.94) gives at(n,m) + a3(n,m) + a3(n,m) + a3(n,m) = 0mod 2*, whence 
by hypothesis we get (12.88) for k = 2 and the first half of (12.88) for k = 3, 

When k=3 we have 
((a1(n,m) + a2(n,m) + a3(n, m) + aa(n,m)) /2)” = (ai(n, m)/2)” + (ar(n,m)/2)” + (ar(n,m)/2)’ + (ar( 
, whence using (12.82) we complete the proof of (12.88). 


We first note a simple special case (91 = Y2 = 93 = Ys = 1, k = 2,3). 
Corollary 12.8.1 If the ring R satisfies the (strong) four squares property (12.80), then so does the polynomial ring 
R[x1, 22,..., £p]. In particular this is true of Z|x1, £2,..., Ln]. 
Corollary 12.8.2 If a,b€ Zlui,u2...ux] can be written a=aa'mod4, b=£6'mod4, with 
a, a’, 8, B' € Z[u1, uz... ux] satisfying a = a' mod2, B= B' mod2, and y1 = af, y2 = —a'B, p3 = —af’, 
ya = a'f' satisfy the hypotheses of Lemma 12.8.1, then for R,S,T,W © Z[ui, ua... ux, 
R? — aS? — bT? + abW? = 0mod4 > R=aS = bT = abW mod2 


(12.96) 


and ifa,b 2 0mod 2, 


R? — aS? — bT? + abW? = 0mod8 = R=S=T=W =O0mod2 
(12.97) 


Proof. Multiplying the left-hand side of (12.96) through by af, and setting r= R, s=aS, t= 6T and 
w = afW, gives the equivalent form 


abr? — a' Bs? — af't? + a’ B'w* = Omod 4. 


(12.98) 


Lemma 12.8.1 then gives (12.96) with k = 2, and (12.97) with k = 3. 


Theorem 12.8.3 Let R be an integral domain of characteristic 4 2, with field of fractions K. Let a,b, a, 8B € R be 


such that a? = amod2 and B? = bmod2, and let 2 be the quaternion algebra oe Let O comprise the 


quaternions of the formr + +c, where c = (af, B,a,1), and r and the components of r are in R, then 


1. O is an order in 2, and N(u) € R for each u € O. If moreover R has the property that 2|2? > 2|x for 
allx € R, then O is independent of the choice of a and f. 

2. If further 

(a) R is integrally closed 

(b) 2 is prime in R 

(c) a and b are not divisible by 2; 

(d) If a and b divide x, then ab divides x. 

(e) If by? — ax”, then b|x, y, and if a\y? — ba’, then a|z, y. 

Q® iif u=(z,y,z,w) € Rt, and = N(u) = 2? — ay? — bz? + abw? = 0mod 4, then 
L = ay = bz = afwmod2, 

then every quaternion in +R with norm in R is in O, and every order which contains i and j lies in O. In 
particular, O is maximal. 


Proof. Clearly O is an ideal. For u=(a,y,z,w) € R‘, a straightforward calculation gives 
uc = cu = («& + ay+ 8z+ abw)emod2 and ec”? = —(b— f”)(a — a”)1 + 2aBe = 2ahe mod 4, from which 
it follows that O is also a ring. Since N(c) = (a? — a)(6? — b) = 0mod 4, it readily follows that N(u) € R for 
uc O. 

If 2|2 => 2|a for all x € R, and a1, 8; € R satisfy a, = amod 2 and Be = bmod 2, then 


(a1 — a)? = (a1 — a)(a1 +a) = 0? — a = 0mod?2, 


so by hypothesis a; =amod2, and similarly 8, = Bmod 2. It follows that 
(a@181, 81,1, 1) = (a8, 8, a, 1) mod 2, so that the definition of O is independent of the choice of a and f. 

Now suppose that (2a)-(2f) hold. If u = (a, y, z,w) € +R and N(u) € R, then (2f) applied to 2u, gives 
22 = a2dy = $2z = af2wmod?. By (2b) and (2c) we may cancel modulo 2 to obtain 2x = 2aG6w, 2y = 26w 
and 2z = 2aw (all mod 2). That is 2u = 2we mod 2, so u € O. (so far using only (2b), (2c) and (2f)) 


Now let O be an order which contains i and j, and suppose v = (x, y, z,w) € O’, then because R is integrally 
closed, tr(v), tr(iv), tr(jv), tr(kv) and N(v) are all in R ({30], Corollary 3.6). These give in turn 2x € R, 
Zaye R, 2bze R, 2abweER and x?-ay?—bz2?+abw*e R. Setting X=27, Y=2ay, Z = 2bz, 
W = 2abw, multiplying the last equation by 4ab gives 


abX? — bY*—aZ*+W* € 4abR 
(12.99) 
whence 
W?—bY2caR W*-aZ’ cbR 
(12.100) 


By (2e) it follows that a|W,Y and b|W,Z, which together with (2d) also gives ab|W. It follows that 
L,Y, Z,w e +R, and since N(v) € R, the first statement then gives v € O. 


Theorem 12.8.3 with R= Z and a=b=-—1, a=8=1 gives Theorem 12.8.2. In this case (2f) is the 
statement that Z has the four squares property. 
Proof of Theorem 12.8.1 . First, we show that any element of @ has half-integer coefficients. Let 
u = (R,S,T,W) € @, let dbe the lowest common denominator of all the coefficients (reduced as far as possible) 
of the components of u, then du € Z[u,v]* and N(du) € d?Z[u, v]. If dis divisible by an odd prime p, then 


21 y21 . 
4 ), which has zero 
Zy(u,v| 


norm, but this is impossible as we will show that J, is a division algebra. By [24], Theorem 2.3.1, it suffices to 
show that the equation 


reducing the coefficients in du mod p we obtain a nonzero quaternion in 2, := ( 


(u? — 1)p*(u, v) + (v? — 1)q? (u,v) = 1 


has no solution with p,q € Z,(u, v). Setting v = 1 this equation becomes (u? — 1)p?(u, 1) = 1, which clearly has 
no solution, as (u2 — 1) is not a square. So we conclude that d is a power of 2. If d were a multiple of 4, then du 
would have integer coefficients, not all even, and norm divisible by 16, but the second part of Corollary 12.8.2, 
witha = uw —1,b =v? -la=u+t+1,8=v+1,a0' =u—1, pf! = v—1, shows that this is impossible. 

To complete the proof, we apply Theorem 12.8.3 with R= Z[u,v], a=u?—1, b=v?—-1, a=u4+l, 
8 =v-+1. In this case, we can easily verify (2a)-(2d). To prove (2e), let u* — 1|p?(u, v) + (1 — v”)q?(u, v), 
where p,q € Z[u, v]. For all v € (—1,1), both summands on the right-hand side are nonnegative. Hence, when 
u = +1, both vanish. It follows that u? — 1 divides p and g. Together with the corresponding statement obtained 
by interchanging u and v, this gives (2e). Finally, the first part of Corollary 12.8.2 gives (2f). 


Lemma 12.8.2 There is a member of u € G, with irrational coefficients. 
An example is the quartic 


u = [(1 — u?)(a — av? + v?) + uv, (v — 1)((b — au)(v + 1) + uv), 
(1 —a)(1 —v)(1— u?) + u,a4 bv — u(a — 1)(v—1)], 
which has norm 1 whenever 2a — 3a? — 6? = 1 — 2a +. a? — ab = 0. A routine calculation shows that these have 


real solutions a=b=1/2, and where a and b are the (unique) real roots of 20° — 227+ 22 —1 and 
2a? + 6x” + 4x — 1 respectively. These roots are not rational. 


12.8.1 Generation 


Here we consider the question as to whether or not Y finitely generated. We thank Alan Reid for providing us with 
a simpler proof than our earlier argument based on arithmetic Kleinian groups. 


Theorem 12.8.4 The group Y is not finitely generated. 


Proof. We may identify Y is the obvious way with the group of elements of norm 1 in the quaternion algebra 


(12.101) 


Suppose that O is a (maximal) order in Yg and simply specialize u,v as follows. Put u = 0 and, for p a prime 
p23, 


am +(p+ Vp? — 4), 


which has conjugate 0 = +(p yp? — 4) € (—1, 1). Then Qg has homomorphic image 


—1,4(p?—2+pv/p?—4) 
Q(/p?—4) , 
(12.102) 


Apart from the identity, the other real embedding is o(v) = 0 and so o(v? — 1) = 0? — 1 < 0. Hence the group of 
elements of norm | in the order O so specialized is some arithmetic Fuchsian group coming from a division 
algebra over 2g(v), see Theorem 12.9.1 below. 

Now the rank of this group must go to infinity with p as there are only finitely many arithmetic Fuchsian 
groups whose quotients are surfaces of a given topological type, [24]. In particular, this implies the group of 
elements of norm | of Ocannot be finitely generated. 


Calculation shows that the 5-element set 


{(u, 1,0,0), (v,0, 1,0), (u,v,0,-1), $(1+utv—w,v—1,1—u,-1), 
$(l+u+v—w,v—1,1-4,1)}, 


(12.103) 


each of which is of degree 1 or 2, generates every member of Y of degree at most 4. However, we also have for 
example (proof omitted) that u below does not lie in the subgroup generated by these elements. 


u = 5(-14+ u? — 2u3 — v? + 8u20? + Qu3v?, 1 — ut Qu? — v2 + Suv? — 2u?0?, 


1—u?+u—2uvt u?v 4+ 4u5v, 1+ u—v + 3uv — 4u72) 


12.8.2 Quaternions as Isometries 


We now look at what happens to the members of Y under the evaluation map 7),),. We will assume for the 
moment that A, . ¢ [—1, 1] and A’ = —1, and abbreviate #1, to y. We set C = I'(A, w) = ¥(Z). Now (12.69) 
and (12.70) become 


a= d=./ 1— 4p, c=-b=—Vl-p 


(12.104) 
and in addition 
ab = cd = 4/1 —py/-1 =t+$V/w-1, 
and so #((R, S,T, W)) is 
Ro + SwVr? —1 +(Ty + Wad? — 1) Vp? -1 
+(Ty — WV —1)\/? —-1 Ry — SyVN—1 
(12.105) 


First we revisit the three quaternions w;€ % (¢=1,2,3) defined at (12.54), which have images 
Ww; := p(wi) € ¥, namely 
wi = (u, 1, 0, 0), 
Ww. = (u, v,0, —1), 
W3 = (1 ae uv,v—1,1—4u,-1), 


(12.106) 


As we have already seen (or directly from (12.105)), these map respectively to the isometries f, gf*g~! and 
[g,f], where X(f) = A, u(f, 9) = w f is loxodromic (since A ¢ [—1,1]), ax(f) = (0,00), and g is an order 2 
elliptic whose axis is disjoint from ax(f) (since » ¢ [—1, 1]), and has mutually reciprocal endpoints. As noted in 
the remarks after the proof of Theorem 12.4.1, this means that the common perpendicular of ax(f) and 
ax(gf?g 1) has endpoints +1. 


We have now got back the subgroup of comprising the balanced even words in fand g (since g is order 2, the 
distinction between regular and irregular words now vanishes). We can now extend this subgroup. Let y ¢(z) = —z 
, pr(z) = 1/z (z) = —1/z; these three isometries are each of order 2, have mutually orthogonal axes and 


b 
generate a Klein 4-group, K. We define h = gyy. Recall (12.68) that g has matrix representative B = ? A 
c 


Thus, using (12.104), A and h? have respective matrix representatives 


M,=-— vutl tV/u-1 id tp? —1 
V4 /u-1 Jeti Le ai 


The axis of A has endpoints +1, and A(h) = py. Also h? € I; specifically h? = o((v, 0, +1, 0)). 

At this point a certain symmetry between f and hf is becoming apparent. Both are loxodromic, both have 
squares in I’ and their axes are mutually perpendicular. To develop this symmetry further we express h, like f as a 
product of two order 2 elliptics. Set g = fp; explicitly, g(z) = A/z, where f(z) = Az, so that g has order 2. We 
now have 


| 


IPF = h, 90n = f, 
A(h) = HF, 9), Af) = wh, 9); 
ax(py) = ax(f), ax(~p) = ax(h). 


We can summarise all this by saying that the pair (h, g) is obtained from (f, g) (up to conjugacy) by interchanging 
the parameters J and uw. 


Theorem 12.8.5 The subgroup P of (f,h) comprising the isometries of the form f™h™ f™h™ ... f™h™*, where 
nytng+...ng andm,+m2+...mz, are both even, is a subgroup of I’. 


Sketch of Proof. We first show that P = c- he the. hf?h-'). This can be done using induction along the 
same lines as the proof of Lemma 12.5.2. We have already seen that f?,h? € I’. The proof is completed by 
showing that fh? f+, hf?h | have respective matrix representatives 


M |" eS 
fhefo = 
+(A — VA? aye 1 Le 
= ((v, 0, tu, +1)) 
pala zVi 1 | 
My pena = 
tV NM? — 15/2? —-1 X- pW? 1 
= v((u, v, 0, +1)). 


Clearly P is a finite-index subgroup of (f,h), and it follows in particular that if I is discrete, then so is (f, h). 


Further (see (12.105)) a sufficient condition for this is that 1 and u both lie in a discrete subring of C (i.e. a subring 
of the ring of integers of some imaginary quadratic field). 


Corollary 12.8.3 If R is a discrete subring of C, f and h are non-parabolic non-identity isometries in Isom* (H”) 
with perpendicular axes, and X(f), A(h) € R, then (f, h) is discrete. 


In particular we have discreteness when X(f),A(h) are integers. Another discrete example is 


A(f) = A(h) = = which minimizes max{r,,7;,} among all two generator non-elementary groups having 
loxodromic generators with perpendicular axes, [23]. We discuss discreteness criteria further in Section 12.9. 
Additionally, we can add in all the order 2 elliptics and preserve discreteness. These elliptics fall into three 


Klein 4-groups: K := {pys, pn, 9}, Ks := {95,9 G95} and Kn = {Pn 9, 99n}- 
Theorem 12.8.6. The group P; generated by KK; and K,, is an extension of (f,h) of index at most 2. 


Proof. Every a € P; can be represented by a word in {y +, Yn, 95959, f,h} which we suppose to have the fewest 
possible elliptic letters, and with the first elliptic letter occurring as close to the right as possible. Ifa ¢ KUKy 
then afa=-f*', so that af=ftta, and similaly af-'=ftla. If a=g _ then 
af =7f" = a ysf*'p;")o'g =hf+t'h-1g, because f and yg; commute. By our assumptions about the 


word, it follows that no elliptic letter can immediately precede an f*!, and symmetrically it cannot immediately 
precede an h* either. It follows that all of the elliptic letters are at the right of the word. But the product of any 
two elliptics is either another elliptic or a product of (at most two) of the loxodromics f+! and h+!, so the word 
contains at most one elliptic letter. Thus P, is an extension of (f,h) of index at most 2. 


12.9 Arithmeticity 


In this section we first recall some further terminology concerning quaternion algebras with an aim to extending 
the discreteness conditions described above. This section is adapted from Section 4 of [9]. 


Let k be a number field. A placev of kis an equivalence class of valuations on k. Such a place is real (complex) 
if it is associated to a real embedding (conjugate pair of complex embeddings) of k. We denote by k, the 


completion of k at the place v. If Q is a quaternion algebra over k, we say that Q is ramified at v if Q @, k, is a 


division algebra of quaternions. Otherwise v is unramified. If v is a real place, then Q is ramified if and only if 
Q ®; ky = H. It is straightforward to check whether a quaternion algebra Q = (3) is ramified at a real place v; 
if v corresponds to the real embedding o, then Q is ramified at v if and only if o(a) and o(b) are both negative. 


We can now define an arithmetic Kleinian group. Let k be a number field with one complex place and Qa 
quaternion algebra over k ramified at all real places. Next let p be an embedding of Q into SL(2,C), let O be an 
order of Qand O, the elements of norm 1 in O. Then p(O;) is a discrete subgroup of SZ(2,C) and its projection 


to PSL(2, C) is an Kleinian group. Kleinian groups so constructed, together with those which are commensurable 
to them, are arithmetic. We note in passing that arithmetic Fuchsian groups arise in a similar manner. However, in 
that case, the number field is totally real and the algebra ramified at all real places except the identity. 
For a subgroup T' of SL(2, C) the invariant trace field is defined as 
kl’ = Q({tr2(g) : g € I}) 
(12.107) 


Then we set 


ar ={Soaigi: ai € Q(tr(P)),9: « T} 


Then QT is a quaternion algebra over Q(tr(I’)). Additionally, if tr(I’) consists of algebraic integers we see that 


Or = {rag 144 © Revra)), Gi € r} 


is an order in QI’. Here Roeviz(,)) is the ring of integers in Q(tr(I’)). Then I is arithmetic if and only if the 
following conditions are satisfied: 
1. kT is an algebraic number field; 
2. tr(I’) consists of algebraic integers; 
3. for every @isomorphism o : kI’—> C, other than the identity or complex conjugation, o(tr(I’))) is 
bounded in C. 


In practice, it is hard to apply this characterization directly, the problem being to establish the boundedness of 
the traces at real embeddings. However, in [9] we obtained the following more useful method for proving groups 
discrete. 


Theorem 12.9.1 LetI be a finitely generated non-elementary subgroup of SL(2,C) such that 


1. kI’ has exactly one complex place or is totally real; 
2. tr(I) consists of algebraic integers; 


3. QV’) is ramified at all non-identity real places of kI, 
then I’ is a subgroup of an arithmetic Kleinian or Fuchsian group. 


Corollary 12.9.1 A group of elements Y of norm 1 in an order @ of the quaternion algebra 2g, 
— ( W-1y?-1 
29 = ( Q(u,v) ) 
is a discrete subgroup of an arithmetic Kleinian group if 


e u is a complex algebraic integer with irreducible polynomial of degree n, which has n — 2 real conjugates, 
11, 12,--+;Tn—2, all of which lie in the interval (—1, 1). 
¢ vis an algebraic integer in Q(u). 
« for each non-identity real embedding o;: Q(u) > Q, o|Q=id, defined by oj(u) =1r;, the image 
o(v) € (1,1). 
Proof. The first condition gives Q(u) a number field of degree n over @ with one complex place, and v € Q(u) 
then gives Q(u,v) = Q(u). If o; is a real embedding, then o;(u? —1) =o;(u)? —1= r?—1<0, and 


oi(v? — 1) < 0 by hypothesis, so the quaternion algebra is ramified at all the real places. Next, the trace is 
2R(u,v) € Z(u, v), which must be an algebraic integer since both u and vare. 


Li 
12.10 Discreteness: Necessary Conditions 


Let I := ,y',(Y), a subgroup of SL(2, C) which, we recall, comprises the matrices of the form 


R(A, #) + S(A, p)V? — 1 2ab(T(A, w) + W(A, p)Vr? — i] 
2ed(T(A, w) — W(A, nw) Vd? — 1) ROA, ) — S(A, nw) VW? — 1 


where the polynomials R, S, Tand Ware polynomials with half-integer coefficients satisfying (12.79), with 


a= +-(VNF1+ VW—2), d= + (VN41- YN =n), 


and band c are given by (12.19) when » = 1, andc = —b= ae otherwise. 
We know that I extends the group of regular even balanced words in A and B given by 


At+V=1-1 


SS —  — 0 
A= V2/X-1 B= (5 a 
0 —r\+VX2—141 cd 
V2Vrx-1 


(12.108) 


As we have observed (Theorem 12.5.1) this group is an index two subgroup of the group of all regular 
balanced words in A and B. Similarly we can show that Tan index two subgroup of the group iy generated by T 
and A. To see this, it is enough to show that, forG ¢ I, AGA! € I, a straightforward calculation using (12.11). 
It follows in particular that I" is discrete if and only if Tis. 

Another routine calculation, using (12.14) and the facts that 8(A) = 2(A — 1) and 


A4abed = be(1 + be) = 4(45+)(1+ 4) = - 1, 
gives 


y(A,W) =tr[A,W] — 2 = —8(A — 1)abed(T?(A, nw) — W7(A, w) (A? — 1)) 
= =2(d = 1)(u? = 1)(T7(A, mw) — W200, 0)? = 1) 


Since R? — (A? — 1)S? — (uw? — 1)(T? — (A? — 1)W?) = 1 we then have 
(A, W) — B(A) = —2(A— 1)[R?Q, n) — A? — 1) 5°, 4)] 
We write 6 = B(A) and ¥ = 7(A, W) to obtain the following three identities: 


B+ FA = [A= 1]. + J = 1], @) — W20A, Hw)? - 1))) 
ll+ 18-71 =2/A-1]0 + |?) — 0? -)8?0,m)) 


and |¥||8 — ¥| 
= 4|A ~ T|? jp? _ 1| |R?(A, L) = (? — 1)S?(A, »)| |T?(A, 1) W?(A, w)(A? a 1)| 
The three equations above enable us to use the following test for the discreteness of r (and so of I). 


Theorem 12.10.1 Let co = 2 —2cos(m/7) + 0.198062 and \7,u2 #1. Then with the notation above, I’ is 
discrete if and only if for every W € I the following three inequalities hold. 


|B|+\F| 21, ify #0,and 


(12.109) 
(12.110) 

MIF — Bl = CO, if y 4 0, B. 
(12.111) 


Proof. |=] First suppose {W;}°°, C I is an infinite sequence, that W; — id as i — oo, and that (with the obvious 
notation)  y;40,8. Then of course ultimately the last inequality is violated since 
i = 7(A, W;) = tr[A, W;] — 2 > 0. To remove the assumption that -; 4 0, 8 we consider two cases. 

Case 1. y; =0 for infinitely many 1. Then W; has a fixed point in common with A in C. Now 
X = BAB € I (only if Bis order 2). If X shares a fixed point with A, or maps one fixed point to another, then 


A and XAX~! have acommon fixed point in C and hence 


0 = (A, X) = 7A, BAB") = 7(A, B)(7(A, B) — B) 


so 7(A,B) =0 or 7(A, B) = 8. However, uw = 1— 27/8 € {+1} in either case, and this is excluded by 
hypothesis. We now deduce that X~!W;X does not share a fixed point with A for infinitely many i, and 
XW;X7! > id. 

Case 2. -y; = @ for infinitely many 7. Then V; = W,AW,! + A,0 = (A, Vi) = 7(A, A7!V;) and so we reduce 
to the first case by replacing W, by [A~!, W;] — id. 

[<=] Next, suppose the group I is discrete, but one of these inequalities is violated for some W € I’. The first 
two inequalities are J@rgensen's inequality and a well known variant of it [10]. These are necessary conditions for 
the discreteness of the group (A, B) provided this group is not virtually abelian. The last condition is a result of 
Cao [3] improving other versions of inequalities Jorgensen found [11, 18] for discrete groups generated by two 
elements of the same trace. We state this in the following lemma. 


Lemma 12.10.1 If (f, 9) is Kleinian and B(f) = G(g), then |y(f,g)| > co where 


co = 2 — 2cos(7) 


This bound is sharp and achieved in the (2,3, 7)-triangle group. 


Thus the violation of one of these inequalities shows that (A, W) is virtually abelian. If the group is abelian, 
then y(A, W) = 0. If WAW! = A“1, the dihedral case, then WAW1A~! = A~?, and hence 


4(A,W) = tr(WAW~!A-1) —-2= trA? —2 = tr?(A) —4= A(A) 


By hypothesis A is not parabolic (A # 1). If A is loxodromic, then (A, W), being discrete, is Kleinian unless W 
fixes or interchanges the fixed points of A. Otherwise there would be three, and hence uncountably many limit 
points, [2]. These reduce to the cyclic or dihedral cases. If A is elliptic, then |G] < 1 is required to violate either of 
the first two inequalities. That is A has order 7 or more. The classification of the elementary discrete groups [2] 
shows this to reduce to the abelian or dihedral cases as well. What remains is the case (A, W) is a discrete group 
with the last inequality violated. Then this group is elementary and as 7(A,WAW~?) = yw(Yw — 8) a little 
argument using the classification of the elementary discrete groups reduces to the previous cases. 


| 
12.11 Examples 


We calculate some of the polynomials for balanced, even, good words in fand g, and investigate when these have 
the same axis as £ For W = fof°g ‘fgf?g ‘f°, 


Qr(u,v) =—143u—2u? — 10u? + 4u4 + 8u° — v — 3uu + 8u?u + 4u3y 
— 8u4u + 2v? — Gur? — 6u2v? + 14u3v? + 4utv? — 8uev? 
2s(u,v) =—1+2u+ 10u? — 4u3 — 8u4 — v — 4uv + 4u7v + 8usv + ur? 


— 6u?v? — 4u3v? + 8utv? 
2t(u,v) =(u—1)(-1+4 2u + 4u?)(-1— 2u + 4u? + 4u3 — 4uv + 4u3v) 
2w(u,v) = (1+ 6u — 4u? — 20u? + 8u4 + 16u° — 2u + 6uv + 8u?v — 20u3v 
— 8utv + 16u°v) 
The only solution of ¢(u,v) = w(u, v) = 0 is u = —1/2, v = —1/3. For these values we also have s(u,v) = 0 


and r(u, v) = —1, ie. Wis a relator of (f, 9) for these values. However, we return back to (12.65) and (12.66) to 
see 


p=2u-2=-3, y=A(1—v)/2=-2 


so fhas order three and fand g. 


Corollary 12.11.1 Let I be a Kleinian group and f,g € I’. Then 


fof?g'fgf'g f° =1 
if and only if f has order 3 and (f, g) is a Euclidean triangle group or an abelian group. 


Of course if fand g commute then fgf*°g 'fgf?g tf? = 1, then f® = 1. 
For W = fof?g tf” 


Qr(u,v) = —1—3ut Qu? + 4u3 — ut 3uv + 2u2y — duty 
Qs(u,v) =1+2u—4u? —v + 2uv4 4u7v 

t(u,v) =(u—1)(1+2u)(—1+ 2u+ 4u”) 

2w(u,v) =(-1+ 2u)(—1+ 2u + 4u?) 


This time ¢ and w have a common factor —1+2u+4u?, so that they vanish simultaneously when 
u = 1/4(—1+ V5), and for all values of v. However, s(1/4(—1 + V5),v) = 1/2(-1+ V5) 40, so W can 
never be a relator of (fg). 

For W = fofg'f'afg fof ‘9 ‘fof ‘9 


r(u, v) =—u-wtututtud — v2 4+ uv? — u?v? + ude? + 2ute? 
— 2u5v? + v4 — 38uv4 + 2u?u4 + 2u3ut — 3u4v4 + ubv4 
s(u,v) =—-1+ uw? + Qu? + ut + vu? — Quy? + 3u2v? — uty? — vt 


+ 2uv4* — 2u3y4 + utv4 
t(u,v) = (u—1)(u+1)(u—v+4+uv)(-1-—u—u?—v+uv4 v? 
— 2uv? + uv) 
w(u,v) = (u—1)(-u— uv? —u3 — v — uv — 20 — u80 4 0? — uv? — u?0? 


+u3y? + v3? — uv? — u?v3 + u3v3) 


Here Wis a relator in the group that minimizes the maximum of the two translation lengths max{r,;, T,}, when f 
and h are two loxodromics with perpendicular axes [23]. Now tand w have a common factor (wu — 1), so that they 
vanish simultaneously when u = 1. However, t = w = 0 also holds when v = 0 (perpendicular axes), when u = 0 


and when u = —1 + iV3. In the last two cases r = 1 and (consequently) s = 0. 
These examples raise some general questions: 


1. Which words can be relators? (ie for which words do s = t = w= 0, r = +1 have a solution, apart from the 
trivial solutions u = 1 (f = Identity) and v = 1 (fand g have the same axis)?) 


2. For which words do t and w have a non-constant common factor in which neither of the variables u and v is 
absent? (giving an infinite family of solutions for t = w = 0) 


3. For which words do t and w have no such common factor, so that £ = w = 0 has only finitely many roots, 
and one of these roots also makes s = 0 and (hence) r = +1 (ignoring the trivial cases u = 1, v = 1). Is any 
such group discrete? 


12.11.1 Explicit Formulae 

We now give (without going into details of the computation) explicit values for the polynomials R = Ry, S = Sw, 
T =T, and W,, of (12.71) associated with the even word w = f"gf™g 'f™gf™g | f", which are expressed 
in terms of Chebyshev polynomials indexed by various combinations of the powers n,. For such a word, and for 


S C 414 2, 3, as 5}, we let Ts(u) = Teyny+€gmyt+€gng-+€anat ess) /2 (u), Us(u) = Veen +Egng+€3n3+€yngt€sns)/2 1(u), 


where €; = —lifie S,e€;=1ifi¢ S, e.g. Tro 3)(u) = Te,—ny—ng-ingtns)/2(¥). (We set U_i(ax) = 0, and, for 
n <0,T,(x) = Ti\(x), Un_1(x) = —Ujnj-1(2)). 
We have calculated: 


R(uv) =F{ So (-2!"'T5(u) } v? 
SC{2,3,4} 


+ 4 (%(u) - T 2,4} (u))v 
+ $(To(u) + Ty (u) + Tes} (u) + Ty (uw) + Ta} (u) + T2,3,43(u) 


S(u, v) is the same, but with Ug substituted for T; throughout in (12.11.1) above, 


wl 


T(u,v) = S> (-1)'sug53(u) | v 
SC{2,3,4} 


+ $(T3,5}(u) — Ta33(u) + Ty53(u) — Tray (u) 
— Tr12,33(u) + T43,4,5}(u) — Tray (u) + Ty5}(u)). 
W(u,v) =+4 (-1)'*'Tsu45(u) } v 


SC{2,3,4} 
+ $(Uq33(u) + Uys5}(u) + Ua ay (u) + Uya5}(u) 
+ Ugi233(u) + U43,4,5}(u) + U3 (u) + Uys} (u)). 


Remark: These formulae exhibit the general fact that if the sequence (n1, n2, 73, 24, M5) is reversed, the sign 
of tis changed, and r, s, w are unchanged. 
We thus have, for the shorter word A“ BA™ BA™ (nq = ns = 0) 


R(u,v) = 5 [(Ten+-netns)/2(t) - Tiny —ng-+nq)/2(U))¥ 
7 (eyemeeeey (u) + T(ny—ng-+ns)/2 (u))] , 
S(u, v) the same, but with U,_1 substituted for T,, throughout, and 
T(u, v) = + yee (u) _ Thess nga (u)|, 
(12.112) 


W(u, v) the same, but with U,,_1 substituted for T, throughout. 


SSSA 
12.12 Roots of Trace Polynomials 


The purpose of this section is to establish a theorem which shows that the zero sets of the “good word” trace 
polynomials discussed in Section 12.5 are dense in the complement of the space of discrete and faithful 
representations of Z,*Z2 for 3 < p < oo. Indeed we show that the complement of the representations which are 
discrete and free on marked generators is the Julia set of the semigroup of good word polynomials, where we 
define the Julia set of any family Y of analytic functions mapping an open set U C C into itself, as the set of 
z € U such that Y is not a normal family in any neighbourhood of z. 


Theorem 12.12.1 Let fg be Mobius transformations with B = B(f) 4 —4, B(g) = —4, y = y(f,g) and suppose 
that (f,g) is not discrete and free on the two generators f and g. Then for any open set U, y € U € C there is a 


good word w = w(f,g) for which the polynomial qu(z) = pw(z, 8), given by Theorem 12.5.3, has a root in U. 


Proof. Since g is of order two, every member of (f, g) can be represented as a good word in fand g. There are two 
cases. 


12.12.1 (f,g) is discrete but not free on generators 


In this case there is a nontrivial good word w € (f, g) representing the identity, whence 0 = 7(w, f) = py(7, 8) 
so that y itself is the root of a good word polynomial. 


12.12.2 (f,g) is not discrete 


Let Ube a neighbourhood of y and define the good word polynomial zero set as 
&={zeC: there is a good word w so that p,,(z, 8) = 0} 


In Section 12.5.1 we gave a few examples of good words. From that table we quickly deduce that among many 
other points 


{0,8,1+8,2+B}c &. 


Thus 2 contains at least three finite points. To simplify notation we suppress the # variable in our word 
polynomials. Next, suppose that for some good word v we have p,(U) N & # 0. Then there is a word w € (f,g) 
and z € U such that p,,(p,(z)) = 0. However, we know that the set of good words is closed under composition, 
and py(pu(z)) = Pw*y(z). Thus z € &. We are left to consider the subcase. 


12.12.2.1 For all good words v, p,(U) 1 & = 0 


Let F = {p, : vis a good word}. We have seen that on UF omits & which contains at least three points. Thus 
Montel's criterion shows that the functions of ¥ restricted to Uis a normal family. In other words U does not meet 
the Julia set of F. Since (f,g) is not discrete, there is a sequence of good words {w,;}*-, in (f,g) with 
w; — identity as i + oo (this convergence is in the topology of PSL(2,C), that is in each entry of representative 
matrices). 


Pw;(Y) = tr[f, wi] — 2 > tr[f, identity] — 2 = 0 


It follows that each neighbourhood of 0 meets p,(U) for some i. We will thus be done if we can show that Julia 
set of Y contains some neighbourhood of 0. 


12.12.2.2 Density of roots near 0 


We analyse this case in a fairly general framework using some of the theory of the dynamics of polynomial 
semigroups. Much more can be found about this subject, see for instance [15, 16, 25, 27, 28] and the references 
therein. The point here is that, for each nonzero 8 = {(f), we can find a good word polynomial which has 0 as a 
repelling fixed point, and another which has zero as a superattracting fixed point under iteration. In such a setting, 
the Julia set of the semigroup generated by these two polynomials contains a neighbourhood of 0 and the 
preimages of 0 are dense in it. 


Lemma 12.12.1 Let p and q be entire functions, with a common fixed point c, which is superattractive for p and 
repulsive for q. Let Y be the semigroup (p,q), then the Julia set of P contains a neighbourhood of c. 


Proof. This is a standard “push me, pull you” argument which we sketch. We may assume that c = 0. We have 
p(z) =az™ + O(z™*?), q(z) = w(z + bz? + O(23)), where a 4 0, m > 2 and |p| > 1. We construct a sequence 
of functions {f,,} inductively by fo(z) = z and fn41(z) = g(f(z)), where g is either p or g. We choose r > 0 to 
be sufficiently small that we can ignore higher degree terms in p and q. Let z) = z be chosen with |z| < r, suppose 
that fo, fi,... fj have already been defined, and set z; = f;(z). If |z,| > 1, then let f,41(z) = p(f(z)); otherwise 
let frii(z) = a(f(z)). As soon as |z,| > 1, the next number z;,;1 is much smaller; then the z; gradually increase 


in size (because |u| > 1), until eventually it exceeds r, and the process begins again. The sequence { z;} is bounded 
above and below |a|r™ < z; < |uIr. 
If fn4i(z) = p(fn(z)), then the logarithmic derivative 


nei (2) 


—= / 
Foal) — P'(?n) aay FG) 


If fn4i(z) = a(fn(z)), then 


fri) _ oar 
fue) 4 (Zn) zn) fn(z) ztbz? fn(z) 


If |Z,_1| > r, then |z,| > |a|r™, and it takes ¢ applications of q to get the size of z; over ragain, where tis at most 


about log),,)(1/( arn), in the course of which we multiply the absolute value of the logarithmic derivative by 
at least 


(1 — |bzn|)(1 — |bznsal)---(1— [bense|) = (1 — |blr)* > (1 = [|r 180/) 


which can be made as close to 1 as we like by taking r sufficiently small. Each time we apply p, we multiply the 


logarithmic derivative by approximately m. It follows that | fle | — oo as n —> oo. Since the | f,(z)| is bounded 


below, it also follows that |f/(z)| — oo as well. Thus no subsequence of {fn(z)} can converge to an analytic 
function. Since | f,,(z)| is bounded above, { f,,(z)} cannot converge to either. Thus (p,q) is not a normal family 
on any neighbourhood of z. 


To complete our proof we recall from Section 12.5.1 the trace polynomials y(y — 8) from the word bab, 
(8—y+1)*7 from the word babab, 7(1—28+72—(8—2)y) from the word baba! and 
(7 — B)(B+4)(B(Y? — 87 — 4) — (y+ 1) + 47? + 47 + 1) from the word ba~*bababa bab. 

The last polynomial here is superattractive at z = 0 and the rest have multipliers at 0 of —8, (1+ g)° and 
1 — 28 respectively, so that for each 6 # 0 at least one of them has z = 0 as a repulsive fixed point. Thus, by the 
lemma, the Julia set of the trace polynomials contains a neighbourhood of 0, and so a zero-free region U has an 
image under a trace polynomial into a region which intersects the Julia set, contradicting the fact that these 
polynomials generate a normal family on U. The case 6 = 0 is separately dealt with in [21]. 
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13.1 Introduction 


In the preface to the first edition of Courant-Hilbert's “Methoden der 
mathematischen Physik” (see [12]), R. Courant noted the danger that 
mathematical research would lose the initial link between the problems and 
methods of analysis and the physical and geometric intuition, the tendencies 
being to refine the methods and to extreme generalize the existing concepts. 

Over the years, these trends led to an increasing distinction between 
pure and applied mathematicians, who severely criticized each other. This 
constructive criticism gave rise to the theory of real numbers and to many 
topological concepts including non-orientable surfaces. It is obvious that 
some areas that use mathematical methods but their object is derived from 
physical and geometric intuition are disadvantaged in such a discussion. 

The present paper is a piece of the bridge between the theoretical 
approach of the pure mathematician and the practical interest of the 
engineer, physicist and applied mathematician. The main purpose is to bring 
together various geometrical and physical concepts relating to surfaces that 
have motivated the development of the theory of Klein surfaces. 

Riemann surfaces, in the form of domains spread out over the complex 
plane were introduced in Riemann's dissertation whose methods were 
developed much further in the first edition of Riemann's paper on Abelian 
functions “Theorie der Abel'schen Functionen” (see [27]), in 1857. 
Riemann's works provided the basic tools to classify all compact orientable 
surfaces and, more generally, to study the topology of manifolds. They are 
equally important for the development of algebraic geometry and the 
geometric treatment of complex analysis. As for the importance that was 
attached to this topic, it suffices to say that Albert Einstein's “general theory 
of relativity” is wholly based on Riemann's ideas. 

In his “Extremale quasikonforme Abbildungen und quadratische 
Differentiale’” (see [35]) Teichmiiller considered the cases of oriented 
bordered Riemann surfaces and non-orientable Riemann surfaces. He 
defined the double of a oriented bordered Riemann surfaces or of a non- 
orientable Riemann surfaces. These are closed Riemann surfaces with genus 
depending on the original surfaces. He introduced the notions of 
meromorphic functions, n-differentials and divisors on bordered non- 
orientable Riemann surfaces. 


Teichmiiller defined, through two examples, the notion of conformal 
invariant for non-orientable regions. It is important to note that according to 
Teichmiiller's definition, a conformal mapping (even in the orientable case) 
preserves only the angles, but not necessarily the orientation. Thus, such a 
mapping is also defined in the non-orientable case. Teichmiiller exhibited 
complete systems of conformal invariants for some special surfaces. For 
example, a simply connected domain with two distinguished points in the 
interior has one conformal invariant, namely the Green function. 

Teichmiiller considered special cases of the fact that one deals with non- 
orientable surfaces by passing to the orientation double cover. To treat the 
most general surfaces, he considered a symmetrization process on the 
corresponding doubles. This idea was used to solve the main problem of 
extremality, known as the Teichmiiller theorem. By using the two-sheeted 
covering (an annulus) of the M6bius strip, he gets the Teichmiiller distance. 
He showed that problems on the projective plane with two distinguished 
points can be reduced to similar problems on the sphere with four 
distinguished points. In two examples of non-orientable surfaces, Mobius 
strip and the projective plane, the problem of finding the conformal 
invariants is lifted to the oriented double cover. Teichmuller considered 
problems on the Klein bottle and lifted them to problems concerning the 
case of the torus. The torus is the two-sheeted orientation covering of the 
Klein bottle. 

The genesis and development of the idea of symmetry are related to 
Lie's and Klein's research that were inspired by their deep interest in the 
theory of groups and in various aspects of the notion of symmetry. 
According to Klein's Erlangen program, a geometry is determined by a 
“domain of action” (the plane, space, etc.) and a “group of automorphisms” 
(or a symmetry group) acting on the domain. When we change the 
symmetry group we change the geometric scheme under consideration, 
namely we obtain a new “geometry.” 

Thus, the main difference between, say, Euclidean and hyperbolic 
geometry is not the possibility of constructing one or more lines passing 
through a point and not intersecting a given line, but the difference in the 
structure of the respective groups of symmetries of Euclidean and 
hyperbolic geometry. Therefore, the object of the geometry is the study of 
those properties of a domain that are preserved by the transformations in a 


symmetry group. The description of all possible geometries is an open 
problem. 

In the same way, classifying non-classical topological compact surfaces 
is the same thing as classifying all orientation reversing involutions of a 
classical compact surface. 

This brings us to the interesting question of the possible global forms of 
various (say, two-dimensional) geometric systems (Euclidean, hyperbolic, 
elliptic) first stated (in connection with Euclidean geometry) by the 
outstanding geometer W. K. Clifford. Today this question is known as the 
Clifford-Klein problem and the possible global forms of geometries are 
called Clifford-Klein forms. 

It is known that there are only two spatial forms of two-dimensional 
elliptic geometry (the sphere and the elliptic plane), but there are as many as 
five forms of two-dimensional Euclidean geometry (the ordinary Euclidean 
plane, the infinite M6bius strip, the infinite cylinder, the torus and the so- 
called Klein bottle). Finally, there are infinitely many forms of two- 
dimensional hyperbolic geometry. 

In this context, non-orientable surfaces are a possible geometric system 
capable of “modelling” the real shape of the universe surrounding us. For 
more details, we refer to the book by Weeks [37], which fills the gap 
between the simplest examples, such as the Mobius strip and the Klein 
bottle, and the sophisticated mathematics found in upper-level college 
courses. 

Spencer and Schiffer in their advanced monograph “Functionals of 
Finite Riemann Surfaces,” extended the investigation of finite Riemann 
surfaces from the point of view of functional analysis, that is, the study of 
the various Abelian differentials of the surface in their dependence on the 
surface itself. 

The methodology that Schiffer and Spencer employed is characterized 
by Ahlfors’ next comment: “such a surface has a double, obtained by 
reflection across the boundary, and one of the main features of the book is 
the systematic use of this symmetrization process.” 

The notion of Klein surface goes back to Felix Klein due to his closing 
remarks in [15], even though one does not find a definition of a Klein 
surface there. Klein surfaces generalize Riemann surfaces, and they are 
dianalytic manifolds of complex dimension |. Roughly speaking, a Klein 


surface is a surface on which the notion of angle between two tangent 
vectors at a given point is well-defined, and so is the angle between two 
intersecting curves on the surface. 

Basic function theory on Klein surfaces and the relation between 
compact Klein surfaces and real algebraic function fields were developed in 
the monograph “Foundations of the Theory of Klein surfaces” (see [3]) by 
N. Alling and N. Greenleaf. They showed that every Klein surface can be 
represented as the quotient of a Riemann surface by a conjugate analytic 
involution. Thus, it is natural to extend on Klein surfaces the most 
fundamental problems in engineering, physics and other sciences. Alling 
and Greenleaf were the ones who introduced the name “Klein surface.” 

Our approach is an alternative theory to the standard theory given by 
Alling and Greenleaf and aims at the natural imbedded of calculus on Klein 
surfaces in the well-known Cartan's model of calculus on manifolds. We 
have developed this theory because of the unusual behavior from the 
analytical point of view of the Alling and Greenleaf's results. For instance, 
functions are not usual functions but equivalence classes of families of 
meromorphic functions relative to dianalytic atlases. Such a family defines 
an usual function if and only if all its members are the same real constant. 
The meromorphic differentials are also equivalence classes of families of 
functions satisfying some compatibility conditions that lead to the 
impossibility of defining a consistent integral on Klein surfaces (see [3, 
Theorem 1.10.4]). 

We follow Schiffer and Spencer's method to study the objects on Klein 
surfaces by means of the complex double, whose existence and uniqueness 
are demonstrated in [3]. 

We are enabled to bring together systematically and concisely the 
concepts of the Green and Neumann functions, the harmonic kernel 
function and the harmonic measure and to build from them an elegant 
generalization for the basic ideas of boundary value problems on Klein 
surfaces. 

The main objectives of study in this paper are the Dirichlet problem and 
the Neumann problem for harmonic functions on Klein surfaces. The 
technique is based on the fact that according to a classical result due to 
Klein, the boundary value problems on a Klein surface can be reduced to 
similar problems on its complex double. This process has many advantages, 


starting from the fact that complex double is a symmetric Riemann surface, 
that is, a Riemann surface endowed with a fixed point free antianalytic 
involution. Consequently, we obtain harmonic functions on a Klein surface 
by adding together a pair of harmonic functions on the symmetric Riemann 
surface, whose singularities lye at symmetric points. In our study, we use 
methods that have wide applicability in function theory and _ partial 
differential equations. 

The symmetric conditions on the boundary determine symmetric 
solutions on the complex double, which lead to solutions for the similar 
problems on the Klein surface. Specifically, in the case of Klein surfaces, 
the formula for the solution of the Dirichlet problem is expressed in terms 
of an analogue of the Green function, which has the symmetry in argument 
and parameter. In these terms, we extend the use of the Green function to 
the study of the harmonic measure on a Klein surface. That is why we 
distinguish the method to solve the Dirichlet problem for harmonic 
functions on a Klein surface, once the harmonic measure on a symmetric 
Riemann surface is known. This procedure generates an explicit formula for 
the solution of the Dirichlet problem on a Klein surface, which is similar to 
the Poisson integral. At the same time, we rewrite the Radon—Nikodym 
derivative of harmonic measure against the symmetric arc length. 

The corresponding solution of the Neumann problem for harmonic 
functions on a Klein surface is expressed in terms of an analogue of a 
Neumann function. 

The harmonic kernel function is related to the classical domain 
functions, such as the Green function and the Neumann function on a Klein 
surface. In such a way it is possible to solve both boundary value problems 
of potential theory on a Klein surface, once the harmonic kernel function on 
a symmetric Riemann surface is known. 

We refer to Krantz [16-18, 20] for an excellent exposition of various 
topics at the interplay between complex analysis and partial differential 
equations. 

The study of objects on Klein surfaces is an important part of surface 
topology due to the applications of these surfaces in several fields of 
science, such as quantum physics, chemistry and biology. Indeed, for a 
physicist, M6bius's band and Klein's bottle are essential elements in the so- 
called annulment of divergences. In chemistry, the recent synthesis and the 


“half-cutting” of a molecular Mobius strip (see [36]) was considered as a 
spectacular event, described as “the most topologically stimulating 
molecular structure synthesized to date” it catalyzed the birth of extrinsic 
graph theory, dealing with topological chirality, a field now burgeoning in 
mathematics. 

The study of liquid crystals is another field where Klein surfaces have 
surprisingly materialized themselves. In the so-called nematic liquids, the 
molecules form ribbons that may or may not be orientable (see [9]). A 
systematic topological analysis highlighted the double topological character 
of distortions in liquid crystals differentiated for “energetic reasons” (see 
[10]). If we consider the potential function of some form of internal energy 
of the ribbons, then the normal derivative on the border characterizes the 
flow of energy across the border. It may be necessary to determine this 
potential knowing the respective flow or the values of the potential on the 
border. These are boundary value problems that will be solved in this paper. 

The natural tendency of some macromolecules to store energy through 
distortions, a fact well-known to chemists, might be the cause itself for the 
formation of non-orientable strings, thus making obvious the practical need 
of dealing with boundary value problems related to them. 

A unified principle for science that works with dualism is presented in 
terms of torsion fields and the non-orientable surfaces, namely the Klein 
Bottle, the Mobius strip and the projective plane, in (see [26]). This 
principle is applied to the complex numbers and cosmology, to non-linear 
systems integrating the issue of hyperbolic divergences with the change of 
orientability, to the biomechanics of vision and the mammal heart, to the 
morphogenesis of crustal shapes on Earth in connection to the wavefronts of 
gravitation, elasticity and electromagnetism, to pattern recognition of 
artificial images and visual recognition, to neurology and the topographic 
maps of the sensorium, to perception, in particular of music. 

As it is noticed in (see [14]), these are the types of problems that 
contribute to a unifying treatment of orientable and non-orientable surfaces, 
not only from the topological point of view (see [11] and [34]) but also 
from an analytical point of view (see [33] and [3]). 


13.2 Klein Surfaces and Symmetric Riemann Surfaces 


Klein surfaces are the most general two-manifolds that support harmonic 
functions. In order to be able to extend results about boundary value 
problems for harmonic functions on Riemann surfaces to Klein surfaces, we 
have to review some results on the topology of surfaces, Klein surfaces, and 
the uniformization of Riemann surfaces. The history of Klein surfaces is 
going back to Klein (see [15]) who considered the group of conformal maps 
of the Klein bottle and other non-orientable surfaces. In their monograph, 
Schiffer and Spencer (see [33]) did the first modern study of the surfaces 
endowed with dianalytic structures. Much of the material of this Section 
will be presented without proofs and will be completed with references to 
proofs. The main reference to topology of surfaces is the monograph of 
Ahlfors and Sario (see [2]). 

A connected topological Hausdorff space 2 is a surface with boundary 


if every point Pe has an open neighborhood U, which is 
homeomorphic to a relatively open subset of the closed upper half-plane. A 
homeomorphism h: 0 — h(() is called a local parameter at the point 
P €U. The boundary 0& of & consists of those points P € 2, such that 
h(P) € R, for all the local parameters A at the point P. The pair (U, h) is 
called a chart. Let h; : U; 3 h;(U;) and h; : U; = h;(U;) be two local 
parameters, such that U;NU;490, i,7¢ I. The mapping hy hy: 
h;(O; 00;) > hi(O,  U;) is called a transition function. 

Let A and B be non-empty open sets in the closed upper half-plane. A 
continuous map of A into B is analytic on A (resp., antianalytic on A) if it 
extends to an analytic (resp., antianalytic) function on some neighborhood 


of A in C into C. If for the complex conjugate of fis analytic on each 
connected component of the set A, then fis called dianalytic on A. 


An atlas of the surface 2 is a family # = { (Gi, h,)|iel \ of charts, 


where (U;);<; is an open cover of 2. The atlas / is dianalytic if all of its 
transition functions are dianalytic. Two dianalytic atlases <& and & are 
called equivalent if & UZ is a dianalytic atlas as well. An equivalence 
class A of dianalytic atlases of 2 is called a dianalytic structure on X. 


A Klein surface is a surface 2% with boundary endowed with a 
dianalytic structure A and will be denoted by xX. Observe that a classical 
Riemann surface is an orientable Klein surface with empty boundary. 

The main tool in our study is the complex double of a Klein surface. 
Details about the history of this concept may be found in [33] and for some 
of its applications see [3], [11] and [34]. 

Let X be a Klein surface endowed with the maximal atlas 


A= { (Gi, h;)|t eI M We recall the construction of the complex double 

of X, which we shall use in order to apply results about Riemann surfaces to 

Klein surfaces. We consider the disjoint union S = UU. Let (Uj, h;) and 
ve 


(U;, h;) be two charts, such that Gag. U; & (). For a point Pet;n U;, the 
set S has two points, which both correspond to the point P, namely the 
point P € U; and the same point P € U ;- We denote the latter one with Pe 


Next, we identify the points P and P*, if the corresponding transition 
function is analytic. If 0X # 0, then there are two points lying over each 
boundary point of X. Identifying these two points, we obtain a surface O,, 


which is called the complex double of the surface X. The two points of O,, 


which lie over the same point of X are called symmetric points of O2. For 
more details, see Alling and Greenleaf [3]. 

Similar to the orientable case it is obtained that the cover group of the 
double cover 7:O2,—> X is generated by an orientation reversing 
involution. For details, see Seppala and Sorvali [34]. 

The next theorem relates a Klein surface to its complex double. We refer 
to [3] for the proof and more details. 


Theorem 13.1. Given a Klein surface X, there exist a double cover 
m:O,— X of the Klein surface X by a Riemann surface O, and an 


antianalytic involution k:O,—- O2, with m*k=7, such that X Is 
dianalytically equivalent with O2/<k>, where <k> is the group generated 
by k. Conversely, given a pair (O2,k) consisting of a Riemann surface X 
and an antianalytic involution k, the orbit space O2/<k> admits a unique 
structure of Klein surface, such that f : OO, — O2/<k> is a morphism of 
Klein surfaces, provided that one regards O, as a Klein surface. 


The mapping z is a local homeomorphism at all points P € Og, for 
which 7(P) ¢ OX. At points lying over the boundary of X, the mapping z 
is a folding map similar to the mapping x + iy > x + i|y| at the real axes. 
For more details about the folding map and the morphisms of Klein 
surfaces, see [3] and [4]. 

By Poincaré's uniformization theorem, each compact Riemann surface 
of algebraic genus g>2 can be represented as an orbit space H/I'* of the 
upper half complex plane H. Next, H is endowed with the conformal 
structure induced by the group M of the Mobius transformations and the 
acting group I"* is a Fuchsian group, that is, a discrete subgroup of M. The 
group I’* can be chosen with no elements of finite order. For details, we 
refer to Poincaré [23]. 

In his unpublished thesis, Preston proved the real counterpart of 
Poincaré's uniformization theorem: for a Klein surface X of algebraic genus 
g=2, there exists a non-euclidean crystallographic (NEC in short) group I’, 
that is, a discrete subgroup of the extended modular group, such that X and 
H/T are isomorphic as Klein surfaces. This NEC group can be assumed 
having no orientation preserving mapping of finite order. For details, see 
[24] and [28]. 

By Klein's definition, a symmetric Riemann surface, (O2,k), is a 
Riemann surface Oz, together with an orientation reversing involution 
k : Og + Og. The involution k is called a symmetry of O2. For more 
details about symmetries of a topologic surface, see [34]. 

A set D of O, is called symmetric if k(D) = D. Thus, given Oa subset 


of X, then 7~1(2) = D is a symmetric subset of Oz. 
A function f defined on a symmetric set is called a symmetric function if it 
is k-invariant, that is, f = f°k. 

Next, we identify X with the orbit space O2/<k> obtained by 
identifying P with k(P), for all P € Og. If U is a parametric disk on _X, 
then 1~!(U) = UUK(U) is a pair of symmetric disks of O,, hence it is 
natural to consider restrictions on UUk(U) for the local study of the 
objects on O,. Since k is an involution without fixed points, one can 
suppose that U 1 k(U) = 0. 


We identify the points of O,, respectively X, with their images on C from 
the corresponding local parameters, with respect to the relation between the 
dianalytic atlas on X and the analytic atlases on O,. Let z be the local 
parameter on U. Then k(z) is the local parameter on k(U) and 
£= (2) = n(z) = m(k(z)) = {z, k(z)} is the local parameter on U. 

Let F(X) be the vector space of the complex functions on the Klein 
surface X and F,(O2) the vector space of the symmetric functions on O2. 
By Theorem 13.1, we conclude that there exists an isomorphism z™* : 
F(X) — F,(O 2), between the vector spaces F(X) and F,(O2). Indeed, let 
F:X—>Cbea complex function on_X, that can take the value ~ only on 
finite sets. Its lifting fto O, is given by 


(13.1) 


Then, it is easy to see that the function 7*, defined by 7*(F) = f is an 
isomorphism. 

Also, to any function g:O2—- C, we can associate a function 
f = 9+4g9 k, which is a symmetric function on O2. Thus, (13.1) defines a 
function Fon xX. 

Let ¥ be a piecewise smooth Jordan curve on a parametric disk 0. The 
curve ¥ has exactly two lifts from 1~!(U). If ¥(0) = 2) = {z0, k(zo)} and 
if y is the lift of ¥ on O, from Z, then k*y is the lift of ¥ on O, from k(zo). 
We refer to [1] for details about covering surfaces. By definition of y, we 
obtain 7 y=a°k°y, hence for any continuous real-valued function F 
defined on ¥, the function f = F'’z is a continuous real-valued symmetric 
function on 7 U k(7). 

The Euclidean lengths of the two curves y and its symmetric k’y, that is 
their lengths with respect to the metric ds = |dz|, may be different. We 
modify this metric and get a new metric do on Oz, such that the lengths of y 
and k°y, with respect to the metric do, will be the same. We define a 
symmetric metric on O, by 


do = 4(ds + ds°k). 


Then the do-lengths of y and k’y are equal. By definition, the length of ¥ is 
the common do-length of y and k*y. Then 


d3(z) = do(z) = do(k(z)),Z = (z) eX 


is a metric on X. The metric d2’ is invariant with respect to the group of 
conformal or anticonformal transition functions of X. 
By definition, 


f Fd = f[ fdo= f fdo. 
y Y key 


For more details about measure and integration on Klein surfaces, see 
[5]. 

Any Riemann surface O, of class C! is endowed with a Riemannian 
metric determined by the line element 


ds = v\dz + pud2|, 
where v is a positive function. If is identically zero, then the metric 
ds(z) = v(z)|dz| 


and the local parameter z are called isothermal. 
It is known that the isothermal metric ds defines a natural analytic 
structure on O,. Similar to the orientable case, the isothermal metric do 


defines a dianalytic structure on the Klein surface X. See [2] for details. 
Next, we give an example of a symmetric isothermal metric (see 
Schiffer and Spencer [33]). 


Example 13.1. The simplest example of a Klein surface 1s provided by the 
Mobtus strip. Consider R > 1 and the annulus 


Ar={zeEClF <|z|< RB} 


of the z-plane. The Mobius strip, denoted by M, is obtained from Ap by 
identifying the points z and —1/z. Let k: Ar > Ar defined by 
k(z) = —1/z. Then (Ar,k) is a symmetric Riemann surface and the 
quotient space Ar/<k> is a Mébius strip. The Mobius strip is obtained by 
cutting the ring along the real axis in the z-plane and joining the two halves 
together along corresponding boundaries. Thus, the annulus Ap with points 


z and —1/Z identified is a canonical form for the Mobius strip. The 
Euclidean metric 


ds = |dz| 


1s not symmetric. We define a symmetric isothermal metric on Ap by 
do = + (|de| + |d(—2)]) 
=F (1 + ar] |dz|. 
By definition, the metric on the Mobius strip 1s 
d3(Z) = do(z) = do(k(z) 


thus, 


The area element da on Ap is 


da(z) = $(1+ tr) amie.) 


lz)’ 


where m is the Lebesgue measure in the complex plane. Then the area 
element d@ on the Mobius strip is 


df (z) = da(z) = da(k(z)), 


hence 


2 
dof(z) = 4 (1 ns 2, ) dm(x,y). 
Let y : a,b] + Ap be a piecewise continuously differentiable curve and let 
f : y({a, 6]) + C be a continuous function. The integral of f on the curve 7, 
denoted by { fdo, is defined by 
7 


J fdo= 3 FOO) (1+ Fie) IY Olet 
and 


[ [Fad =4$ [f@)(1+ 2.) dm(a,y). 


M Ar 


Let y be a o-rectifiable Jordan arc y, parametrized in terms of the arc o- 
length. Therefore, y: z = z(s) = x(s) + iy(s), s © [0,1], where / is the o- 
length of yy. Then the unit inward normal vector to y at z(s) is 


d ; — 
n, = (—<,#) and we denote b —~2 the inward normal derivative, 
do? d Y 9 
o o Ne 


with respect to the symmetric metric do. In this way, our approach is 
consistent with Nevanlinna [22], Bergman [8] and Schiffer and Spencer 
[33]. For more details about the normal derivative and Green's identities in 
terms of do, see [7]. 


ee 
13.3 The Dirichlet Problem for Harmonic Functions 


This section is devoted to the study of harmonic functions with Dirichlet 
boundary condition on a Klein surface. The similar analysis in the complex 
plane has been developed in Krantz [18, Section 1.2]. 

The notion of harmonic function, as being a solution of the Laplace 
equation, makes sense on a Klein surface. Moreover, a Klein surface is the 
most general two-manifold in which this notion of harmonic function makes 


sense. For details, see [3]. We notice that the notion of analytic function is 
meaningless on a Klein surface. 

The Dirichlet problem on an arbitrary Riemann surface can be solved 
because the property that a function that is harmonic remains invariant 
under bi-holomorphic mappings. For the existence of a harmonic function 
that vanishes on the boundary and has a finite number of isolated 
singularities with given singular parts in a relatively compact region, which 
is contained in a chart of a Riemann surface, we refer to Ahlfors and Sario 
[2]. 

Any Klein surface X can be regularly imbedded in a border free surface 
using a duplication process (see [2]). Therefore, for the boundary problems 
involving a part of OX we can consider it as a part of the boundary of a 
region on a border free surface. 

Let O, be a region in the complex plane, bounded by a finite number of 


analytic Jordan curves. Then Og = O.UOQz can be conceived as a 
bordered Riemann surface (see [1], [33]). Because the Klein surfaces X and 
O2/<k> are dianalytically equivalent, a boundary value problem on a 
region 0 of the Klein surface X, can be replaced by a similar problem on a 
symmetric region D of its double O,, as follows. 


Consider the Dirichlet problem on X for harmonic functions 


AU =0on2 
U = Fonds. 


(13.2) 


where () is a region of X bounded by a finite number of o- rectifiable Jordan 
curves and Fis a continuous real-valued function on O92. 

We define D = 1-1(Q) and f = F’x on OD. Then D is a symmetric 
region of Oz, bounded by a finite number of o-rectifiable Jordan curves on 
O,, some of which may contain part of OO2. Since 7°k = 7, we obtain 
f = f° kon OD, hence fis a symmetric, continuous real-valued function on 
OD. The Dirichlet problem (13.2) on X is equivalent with the following 
Dirichlet problem for harmonic functions on O, 


Au = 0on D 
u = fondOD. 


(3.3) 


For details about the Dirichlet problem on bordered Riemann surfaces, 
see Ahlfors and Sario [2]. 

The Dirichlet problem turned out to be fundamental in many areas of 
mathematics and physics. For example, if Dis a thin, heat-conducting metal 
plate and fis a continuous temperature distribution on OD, then the solution 
u of problem (13.3) represents the resulting steady-state heat distribution on 
D (see [12], [19]). 

Using the maximum principle for harmonic functions, it follows that the 
Dirichlet problem (13.3) with continuous boundary values has a unique 
solution for any region D with only regular points. For some basic 
monotonicity, analytic and variational methods of the theory of partial 
differential equations of elliptic type, we refer to [32]. 

The symmetric conditions on the boundary imply symmetric solutions 
for the problem (13.3). For more details, see Schiffer and Spencer [33]. 


Proposition 13.2. A solution u of the problem (13.3) is a symmetric 
function on D. 

Proof. Let u be a solution of the problem (13.3). We define 7 : D > R by 
a = $(u+u’k). Then At =0 on D. By hypothesis, f = f°k on OD, 
hence 


a=4(f+f°k)=f ondD. 


Thus, wu is also a solution of the problem (13.3). The uniqueness of the 
solution yields u = u on D, therefore, u = u°k on D. Oo 


13.3.1 The symmetric Green function 


Let D be a symmetric region bounded by a finite number of o-rectifiable 
Jordan curves on the symmetric Riemann surface Op. 

Fix a point ¢ € D. The function v(z, ¢) = —In|z — ¢| is harmonic at 
all points z ~ ¢. Let w be the solution of the Dirichlet problem on D, with 


the boundary condition w(z) = v(z,¢) on OD. The unique function 
Gp(z,¢) = v(z,¢) — w(z) defined on D\{C} is called the Greenfunction 
of the region D, with singularity at ¢ (see [2]). 

We assume that u and v are continuously twice differentiable in D and 
once on the boundary OD. We will use the following Green formula: 


f (u fe vp \do = —f [(uAv-—v Au)dzdy, 
aD : : D 


where do is the arc o-length element on OD and the derivatives on the left 
are taken with respect to the inward normal on OD. For more details, see 
[22]. 

The next theorem is similar to the Cauchy integral formula for harmonic 
functions in terms of the metric do. 


Proposition 13.3. (Green representation formula) Let D be a symmetric 
region bounded by a finite number of o-rectifiable Jordan curves and let u 
be a harmonic function in D and continuously differentiable on its boundary 
OD. Then, for all € in D, 


uQa=d f (ue) Se - (2,032 Jado, 


(13.4) 


where the derivatives are taken with respect to the inward normal on OD. 


Proof. Fix a point ¢ € D and a positive number é¢ that is less than the 
Euclidean distance of ¢ to OD. Define D; = D\D(¢,e). Let C, be the 


negatively oriented circle of radius ¢, centered at ¢. We apply the Green 
formula for D-, with the harmonic functions u and v. It follows that 


(135) 


The curve —C, is parameterized by z = z(0) = C+ ce’, 0<0<27. We 
deduce that 


27 


du a du(2(9)) 
— JP Pre —do = f V5, do = EA AG) Emi 00 


As the function u has continuous partial derivatives in D, there is a constant 
C such that | 2 <C' on C,.. Then, on C,, we obtain 
dp 


a <27rCe|ln ec}. 


We observe that the right-hand side of the last inequality tends to zero as ¢ 
tends to zero. Therefore, 


Ou 
pe LP do = 0. 


Using the mean value property, we have 
20 


Jan. Ou —do = — J u(z(0))d0 = —27u(C). 


Then relation (13.5) becomes 


The proof is now complete. o 


Following Nevannlina (see [22]), we obtain that the values of u inside 
D are determined from its values and the values of the normal derivative of 
the Green function on the boundary OD. 


Theorem 13.4 


Let D be a symmetric region, whose boundary OD consists of a finite 
number of o-rectifiable Jordan curves. If u is harmonic on D and 
continuously differentiable on OD, then for all ¢ in D, 


u(6) = ae ful) GO do. 


(13.6) 


Proof. Applying Green's formula for D with the harmonic functions u and 
w, we obtain 


(13.7) 


Dividing (13.7) by 27, and adding this identity to the Green representation 
formula, we obtain (13.6). Oo 


The function 


OG p(z, 
ROS 


is called the Poisson kernel of the Laplace operator and the Dirichlet 
problem on the region D. 


We define Gz, ¢) as 
GM (2,2 = 2 [Gn(z,0 + Gn(z,K(O)] 
on D\{¢, k(¢)}- 


Let w, be the solution of the Dirichlet problem on D, with the boundary 
condition ws(z) = +[v(z, ¢) + u(z, k(Q)] on OD. Then 


GD (2,0) = Flv(z, 0) + v(z, b(O)] — w,(2). 


Therefore, Gz, ¢) is a harmonic function of z in D\{¢,k(¢)}, with 
singularities —+In|z — ¢| and —FIn|z — k(¢)| at € and k(C), respectively. 
Also, G®)(z, ¢) = 0 for all zon OD. 

We can derive the following result (see [7]): 


Proposition 13.5. For every symmetric region D, the function eee ¢) is 
symmetric on D, that is, for all z € D, 


GY (z, 0) = GE (kz), ©). 


Consequently, the function GE) (z, ¢) is called the symmetric Green 
function of the region D, with singularities at € and k(¢). 

An explicit form for the symmetric Green function of the annulus is 
obtained in [7]. For additional information on this topic we refer to [33]. 


13.3.2 The symmetric harmonic measure 
Let D be a symmetric region bounded by a finite number of o-rectifiable 
Jordan curves on O, and @(OD) the o-algebra of Borel sets of OD. The o- 
algebra of symmetric Borel sets of OD is denoted by #,(OD) and 
B (OD) = {UUkKU)|U € AOD) }. 

The harmonic measure for D is a function wp : D x &,(OD) — [0,1] 
such that: 

1. for each €¢ € D, the map Btowp(¢,B) is a Borel probability 

measure on OD; 


2.if f : 0D —R is a continuous function, then the solution of the 
Dirichlet problem, for D and the boundary function f is the 
generalized Poisson integral of fon D given by 


Pof (6) = J f(z)dwp(¢, z),¢ € D. 


(13.8) 


For details, see [25]. 


Remark 1. The uniqueness of wp is a consequence of the Riesz 
representation theorem. 


An extensive study of the harmonic measure is developed in [13]. 
A method of determining the harmonic measure is given by the 
following characterization (see [25]): 


Proposition 13.6. The function wp(-, B), is the solution of the generalized 
Dirichlet problem with boundary function f = 1p. 


The harmonic measure for D is related to another conformal invariant, 
the Green function for the symmetric region D. 

Using Theorem 13.4 and the fact that Borel measures are determined by 
their actions on continuous functions, we obtain a representation of the 
harmonic measure in terms of the inward normal derivative of the Green 
function with respect to do. 


Proposition 13.7. Let D be a symmetric region, whose boundary 0D 
consists of a finite number of o-rectifiable Jordan curves. If ¢ € D, then for 
any z € OD, 


dup(C, z) = PC 0(e6) dal) 


Thus, the harmonic measure for ¢ € D is absolutely continuous to arc 
o-length on OD and on OD, the density being 


Let w®) : D x &;(OD) — [0,1] be the function defined by 
k ~ 
wp (6B) = Flen(6,B) + wn(k(Q), B)] 


where ¢ = te Kg), ¢€D,BeE &,(OD); see [29]. 


Remark 2. The symmetry of the region D implies that the function 


wy) (C, B) is symmetric with respect to B on &,(OD), that is, for any B € 
£,(0D), 


wy) (GB) = wi) (G, k(B)). 


The function wi) (¢ , B) is called the symmetric harmonic measure for 
D. 
The function 


is called the symmetric Poisson kernel for the region D. 


13.3.3 The Dirichlet problem on the complex double 


The following Poisson integral formula both reproduces and creates 
harmonic functions on the complex double. Roughly speaking, the next 
theorem yields the formula for the solution of the Dirichlet problem (13.3) 
on a symmetric region D, in terms of the symmetric Green function. 


Theorem 13.8. Let D be a symmetric region bounded by a finite number of 
o-rectifiable Jordan curves and let f be a symmetric, continuous function on 
OD. There is a unique symmetric function u on D, which is harmonic in D, 
continuous on D, such thatu = f on OD. Moreover, for all ¢ in D, 


ag) (z, 
uQ=2 f todo, 6 D. 
OD 


(13,9) 


Proof. By Theorem 13.4, for all ¢ € D, 


uQat fue) Bae. 
OD 


Replacing ¢ with k(¢) we obtain 
OG p(z,k 
u(k(6)) =f u(z) PEMD) do, 


Adding the last two equations and dividing by 2, we obtain 


w(g+u(e() f u(2) | eee + 260(2M¢ | do. 


2 4n 

aD 

for all ¢ € D. By Proposition 13.2, wis a symmetric function on D, then the 
left-hand side of the last equality is w(¢). We conclude that for all Cin D, 


wl) = Af u(z)[2QEO + A6ofeHO) | gy, 
OD 


The uniqueness of the solution of the Dirichlet problem for harmonic 
functions implies relation (13.9). Oo 


Theorem 13.8 is the equivalent of the Poisson formula for the solution 
of the Dirichlet problem on the disc in the complex plane, see Krantz [20, 
Section 7.3]. In such a way, Theorem 13.8 creates a function that agrees 
with fon the boundary of the domain D and is harmonic inside. 

The formula for the solution to the Dirichlet problem on the annulus is 
obtained in [6]. 

In a similar way, we obtain the following representation of the solution 
of the problem (13.3) on a symmetric region D, in terms of the symmetric 
harmonic measure. 


Theorem 13.9. Let D be a symmetric region bounded by a finite number of 
o-rectifiable Jordan curves and let f be a symmetric, continuous function on 
OD. There exists a unique symmetric function u on D, which is harmonic 
on D, continuous on D, such thatu = f on OD. For all ¢ in D, we have 


= 1 Ie 2)du) (Ge): 


(13.10) 


Proof. Let ¢be a point in D. By (13.8), for all ¢ € D, 


ea ae )dw(¢, z) 


Replacing ¢ with k(¢) we get 
u(k(¢)) = f u(z)dw(k(¢), z)do. 


oD 


Adding the last two equations and dividing by 2, we obtain 
MOT = FS Fe) [do(G, 2) + dol R(C), 2)] 


for all Cin D. By Proposition 13.2, uis a symmetric function on D, then the 
left-hand side of the last equality is w(¢) and we conclude that for all Cin D, 


= 140 2)dw) (Ge 


The proof is now complete. o 


By Proposition 13.7, we obtain the Radon-Nikodym derivative of 
symmetric harmonic measure for D against o-arc length. 


Proposition 13.10. Let D be a symmetric region whose boundary 0D 
consists of a finite number of o-rectifiable Jordan curves. If ¢ € D, then for 
any z € OD, 


k) x AG (2,0) do(z 
du) (,2) = Sgee . SO. 


This result shows that the symmetric harmonic measure for D is 
absolutely continuous to arc o-length on OD and on OD, the density being 


QT On, 


diy? 1 0G (z,0) — pl) (z). 


13.3.4 The Dirichlet problem on the Klein surface 


Let X be a Klein surface and let 1 be a region bounded by a finite number 
of o-rectifiable Jordan curves. The Klein surface X is the factor manifold of 
the symmetric Riemann surface O, with respect to the group <k>. Then, 0 


is obtained from the symmetric region D by identifying the corresponding 
symmetric points. 


The Green function of 0 with singularity at ¢ is defined by 


Gol(Zz,0 = G9 (z,0 = EP (Ka), 0, 


where Z = 7(z). 
By definition, the function Gg(Z,¢) is continuous on 2, harmonic on 
At \ and has the singularity at € = 7(C). 


Remark 3. By Proposition 13.5, it follows that Gg(z,¢) is well-defined on 
2. 


An explicit form for the Green function of the Mébius strip is obtained 


in [7]. 
The harmonic measure for 0, wa : 2 x B(OM) — [0,1], is defined by 


wo(€, B) = wy (GB) = wf (6, a(B)). 


for all € € Nand B = 7(B) € A(AN). 
The function 


is called the Poisson kernel for the region 2. 


Remark 4. By Remark 2, it follows that the function wy is well-defined. By 
Proposition 13.5, 1t follows that the function P; 1s well-defined, too. 


The symmetric solutions on O, determine the solutions of the similar 


problems on the Klein surface X. 
Consequently, we obtain the solution of the Dirichlet problem on the 
region (, with respect to the Green function of (2. 


Theorem 13.11. Let F be a continuous real-valued function on the border 
O92. The solution of the Dirichlet problem (13.2) with the boundary 


function F is the function U defined on 22, by the relation u = U° x, where 
m 1s the canonical projection of Oj on X and u 1s the solution (13.9) of the 


Dirichlet problem (13.3) on the symmetric region D, with the boundary 
function f, given by f = F'n. 


Proof. By definition, AU(¢) = Au(¢) = 0, for all ¢ € 2, where ¢ = 1(C). 
Thus, U is a harmonic function on 0. The symmetry of the function f on 
OD implies 


U(¢) = ul) = FC) = FAG) = FO), 


for all ¢ € OM. Due to the uniqueness of the solution, the function U 
defined on 2 by 


U(¢) = u(¢) = u(k(¢)), 


for all ¢ in 2, where ¢ = 7(C), is the solution of the Dirichlet problem 
(13.2) on Q. Oo 


In a similar way, we obtain the solution of the problem (13.2) on the 
region (0, with respect to the harmonic measure for the region (2. 


Theorem 13.12. Let F be a continuous real-valued function on the border 
O92. The solution of the problem (13.2) with the boundary function F 1s the 


function U defined on 22, by the relation u = U°n, where 7 is the canonical 


projection of O, on X and u is the solution (13.10) of the problem (13.3) on 
the symmetric region D, with the boundary function f, given by f = Fn. 


By Proposition 13.10, we obtain the Radon-Nikodym derivative of 
harmonic measure for Q against X-arc length. 


Proposition 13.13. Let Q be a region bounded by a finite number of o- 
rectifiable Jordan curves. If € € 2, then for all Z € 82, 


dwwo(G, 2) = dw) (¢, z) = dw) (G, k(2)). 


This result implies that the harmonic measure for ( is absolutely 
continuous to arc £-length on O02 and on OY2, the density being 


dw ~ 


ee 


13.4 The Neumann Problem for Harmonic Functions 


This section is devoted to the study of harmonic functions with Neumann 
boundary condition on a Klein surface. The similar analysis in the complex 
place has been developed by Schiffer and Spencer [33]. 

Consider the Neumann problem for harmonic functions 


AU =0o0n 22 
oY — Gon ON, 
ns 


(13.11) 


where Q) is a region of X bounded by a finite number of o-rectifiable Jordan 
curves and Gis a continuous real-valued function on 02. 

We define D=7~1() and g=G°n on OD. Since 1°k =7, we 
obtain that D is a symmetric region bounded by a finite number of o- 


rectifiable Jordan curves on O,, some of that may contain part of OO, and g 
is a symmetric, continuous real-valued function on the boundary 0D. 

The Neumann problem on _X is equivalent with the following Neumann 
problem on O, 


Au = 0on D 
f+ = gon OD. 


(13.12) 


Since k is an antianalytic involution, the symmetry of D and the 
symmetry of g on OD, imply that the prescribed values of the normal 
derivative satisfy the compatibility condition 


f gdo = 0. 
aD 


Therefore, the Neumann problem on O, for the region D and the boundary 
function g has solutions. For details, see [21]. 


Proposition 13.14. If the problem (13.12) admits a solution, then it is 
unique up to an additive constant. 
Proof. Let u, and u, be solutions of the problem (13.12). If u = uz — ua, 


then u is harmonic on D and oe = 0 on OD. Applying Green's first 
identity, we get 


Jd (u2 + u2)dady = 0. 


Therefore, u is constant on D. o 


Proposition 13.15. The solution of the problem (13.12) is a symmetric 
function on D. 


Proof. Let u be a solution of the problem (13.12). We define 7: D > R by 


u= x (u + u°k). By hypothesis g = gk on OD, then xe = pe = gon 


OD and Au = 0 on D. Thus, @ is also a solution of the problem (13.12). By 
Proposition 13.14, there is a constant c such that u= w+ c on D. Thus, 
u k=wu-+2c on D and using the symmetry of the region D, we obtain 
u=—u _k+2con D. Hence c = 0, that is, uk = uon D. Oo 


13.4.1 The symmetric Neumann function 


Let ¢ be a point inside D. A Neumann function Np(z, ¢) for the region D, 
with singularity at ¢, in terms of the metric do, is the function 


Np(z, ¢) — (2, ¢) _ A(z, Os ZE D, z # G 


where h(z, ¢) is a solution of the following Neumann problem in terms of 
the metric do: 


Ah(z,¢) =0, zED 
ph (z,¢) = #(z,0)-7#, ze OD, 


where 1 = f do is the o-length of OD. 
aD 


Remark 5. The boundary value of the inward normal derivative of the 
Neumann function 1s a constant equal to ar 


Theorem 13.16. Let D be a symmetric region bounded by a finite number 
of o-rectifiable Jordan curves. If u is harmonic in D and continuously 
differentiable on OD then, up to an additive constant, 


u(¢) =—-b f 2ONp(z,0do, CD. 


On Ong 


Proof. Fix a point ¢ € D and a positive number « that is less than the 
Euclidean distance of ¢ to OD. Define D; = D\D(¢,e). Let C, be the 


negatively oriented circle of radius ¢, centered at ¢. Applying Green formula 
for D, with the harmonic functions / and u, we obtain 


(13.13) 


Dividing (13.13) by 27 and adding this identity to the Green representation 
formula, it follows that 


u(¢) = -x [ N(z,Q gdo + + f ude. 


aD aD 
Thus, u is determined up to the additive constant + f u(z)do. O 
aD 


Let N &) (z, C) be the function defined by 


nw (z,2) = 4[Np(z,¢) + No(z,k(0)], = € D\{C, k(O)}; 


where Np(z, k(¢)) is a Neumann function for the region D, with singularity 
at k(C) and ¢ = {¢, k(C)}. Therefore, 


NY (2,0) = 4 [o(z,0) + u(z,k(Q)] — hslz, 0, 4G 2 FRO, 


where A, is a harmonic function on D that satisfies 


Therefore, N Wz, ¢) is a harmonic function of z in D\{¢, k(¢)}, with 


(i) 
singularities at € and k(¢) and oN (z,C) = 7, for all z € OD. 


An explicit form for the function N Oey, ¢) of the annulus and of the 
MO6bius strip are obtained in [31]. 


Proposition 13.17. If D is a symmetric region, then the function N &) (z, ¢) 
1s symmetric with respect to z on D, that 1s, for any z € D, 


NP(z,0 = NP (K2), 0). 


Proof. Let h*(-,¢) be a harmonic function in D, such that 


(2,6) = (2,0) + HH, 6) — 3, 2€ OD. 


Therefore, 


gh 40) = gh (k(z),¢), for allz € OD. 


On, 


By Proposition 13.15, h*(-, ¢) is a symmetric function. Hence the function 
k * 
My (2,€) = Flvled) + v(A(2),0)] — A230) 


-_ 7 eM), x 
is a symmetric function, harmonic in D\{¢ \ and —2-(z,¢) = 2%. So, 


N Wz, ¢) and M Wz, ¢) are solutions of the same Neumann problem. 
Thus, by Proposition 13.14, there is a constant c_ such that 


N Bez, ¢)=M Oz, ¢) +c. Since M Wz, ¢) is a symmetric function, we 
obtain that NV - (z, C) is also a symmetric function. o 


Let & be a point of D. A Neumann function Np(z, ¢) is not a conformal 


invariant, but the difference Np(z,¢) — Np(z,¢o) is a Neumann function 
and has a vanishing normal derivative on OD, hence it is a conformal 
invariant. We redefine the difference 


Np(z,¢, 60) = No(z,¢) — No(z, 60) 


to be a Neumann function for the region D on the Riemann surface O2, see 
[33]. 


The function N &) (z, C, Co) defined by 


NM (z, €,€0) = $[No(z, ¢, Co) + No(z, k(), k(Go))], 


for all z€ D\16,0}, is called asymmetric Neumann function for the 
region D. 
13.4.2 The symmetric harmonic kernel function 


Let D be a symmetric region in the complex plane, bounded by a finite 
number of o-rectifiable Jordan curves. In this section, we introduce closed 
systems (y;), of harmonic functions in D, which are orthonormal with 
respect to the Dirichlet integral 


0 aan OF a 0 aan dO; 


We recall some notions and results about orthogonal harmonic functions. 
For more details, see [8]. 

Let A?(D) be the set of harmonic functions g in D with a finite 
Dirichlet integral 


Dip} = Diy, p} < co 


such that 


D{Nod(z, 6), 9(0)} = 20(¢); 


where Np(z,¢) is the Neumann function of D with its singularity at the 
fixed point ¢ ¢ € D. 


Remark 6. The second condition is imposed to normalize D{p,y} to be 
zero if and only if g vanishes identically. 


Proposition 13.18. There exists a closed system (p;), for the class A*(D), 
which 1s orthonormal with respect to the Dirichlet integral, that ts, 


D{O;,0;} S03; 07 = 1, 67 S00 Fg. 


Let ¢ be a point inside D. The harmonic kernel function K p(z, ¢) of the 
closed orthonormal system (y;),; for the region D, with respect to the point 


¢, is the function defined by 


Kp(z, ¢) =  aaeer pilz)y:(¢), zeED. 


The harmonic kernel function is uniquely characterized by the following 
properties: 


Kp(z,¢) = Kp(¢, 2) 


and 


D{Kp(z, ¢), p(¢)} = y(¢), pe A?(D). 


An extensive study of the harmonic kernel function is due to [8]. 

The representation of the harmonic kernel function in terms of a closed 
orthonormal system gives the opportunity to solve numerically the Dirichlet 
problem for arbitrarily multiply connected regions. This is an important tool 
in physics, in particular in fluid mechanics, elasticity and electricity. 

It is known that the harmonic kernel function Kp(z,¢), the Green 
function G p(z, ¢) and the Neumann function Np(z, ¢) satisfy the relation 


Kp(z, ¢) = 3-[Np(z, ¢) aa Gp(z, OI, ZE D. 
(13.14) 


We first derive a formula that solves the Dirichlet problem (13.3). We 
prove that if u is harmonic inside a region D and continuous on OD, then 
we can determine the values of u inside of D by integrating on OD the 
product of u times the inward normal derivative of the harmonic kernel 
function for the region D, which is a fixed function that depends only on D. 


Theorem 13.19. Let D be a symmetric region bounded by a finite number 


of o-rectifiable Jordan curves. If u is harmonic in D and continuous on D, 
then, up to an additive constant, 


u(¢) = = u(z) Ape do, ¢ € D. 
'D 


(13.15) 


Proof. From (13.6), the solution of the Dirichlet problem (13.3) is 


uQ) =a ful) Edo, Ce D. 


aD 
(13.16) 
Using (13.14), we obtain 
OKp(z,¢) _ 1 ONd(z,0) _ 1 Ogp(z,¢) 
Ong 2m Ong Ir Ong 


for any z € OD, where / is the length of 0D (see [21]). Combining this 
relation with (13.16), we find 


u(Q) = — f u(z) RE2do+ 4 f u(z)do. 


oD ’ aD 


Thus, u is determined up to the additive constant + f u(z)do. o 
aD 


Next, we derive a formula that solves the Neumann problem (13.12). 


Theorem 13.20. Let D be a symmetric region bounded by a finite number 
of o-rectifiable Jordan curves. If u is harmonic in D and continuously 
differentiable on OD then, up to an additive constant, 


ul)=— f G2 Kp(z,0)do, CD. 
OD 


(13.17) 


Proof. By Theorem 13.16, using Green formula, it follows that, up to an 
additive constant, a solution of the Neumann problem is given by 


u() = —ae S GE Npl2, Odo, ¢€ D. 


Qa 0 
aD 
(13.18) 
The constant is chosen such that u(z) is in A?(D). 
By (13.14), for ¢ € OD, we have 
Kp(z, ¢) ah = Np(z, ¢): 
Substituting this in (13.18), we obtain (13.17). oO 


Let K “ ) (z, C) be the function defined by 


Ki) (z,0) = $[Ko(z,0) + Kn(z,k(¢))], 2 € D, 


where Kp(z,k(¢)) is the harmonic kernel function of the closed 
orthonormal system (y;),;, for the region D, with respect to the point k(¢). 


The function K \*) (z, €) is in A2(D) (see [8], [30]). 


Proposition 13.21. If D is a symmetric region, then the function K . ) (z, C) 
is symmetric with respect to z on D, that is, for every z € D, 


KP (z,0) = KP (k(2), ©. 


Proof. We use (13.14) and the symmetric properties of the symmetric Green 
function and symmetric Neumann function. Oo 


Let Cy be a point of D. Let A?(D) be the class of harmonic functions @ 
that satisfy the conditions: 


D{y, p} < 


and 


y(Co) = 0. 


The harmonic kernel function K p(z, ¢, Co) of the class A?(D) is related to 
the harmonic kernel function K p(z, ¢) of the class A?(D) by the following 
identity: 


Kp(z, G, Co) = Kp(z, ¢) ~~ Kp(¢, Co). 


The harmonic kernel function K p(z, ¢) for the region D, with respect to 
the point ¢ is not a conformal invariant but the harmonic kernel function 
Kp(z,¢,¢o) is invariant under conformal mapping (see [8]), therefore, 
Kp(z, ¢, Co) is well-defined on the Riemann surface Oo. 


The function K . ) (z, C, Co) defined by 


Kz, ¢,€0) = $([Kp(z, 6, C0) + Kp(z, k(0), k(G0))), 


for all z € D\{é ‘ G ie is called the symmetric harmonic kernel function for 
the region D. 


13.4.3 Integral representations on the double cover 


We first express the solution of the Neumann problem (13.12) for harmonic 
functions in terms of do as a line integral involving the boundary function 
and a symmetric Neumann function. 


Theorem 13.22. Let D be a symmetric region bounded by a finite number 
of o-rectifiable Jordan curves and let g be a symmetric, continuous function 
on OD. If u is harmonic in D and g is its inward normal derivative on OD, 
then up to an additive constant 


ul) =-£ fo(2)N5(z, Odo, Ce D. 


(13.19) 


Proof. By Theorem 13.16, up to the additive constant + f u(z)do, we have 
oD 
for all ¢ € D, 


Adding the last two equations and dividing by 2, we obtain, up to the 


additive constant + f u(z)do, 
aD 


uic)rulh()) pf g(z) NoleS)+NoleM(®) gy 
, 


oD 


By Proposition 13.15, u is a symmetric function on D, then the left-hand 
side of the last equality is u(¢). Therefore, 


u(¢) = —a J 9(2)[Np(z,¢) + Np(z, k(¢))|do, 


up to the additive constant + f u(z)do. Oo 
aD 


Similarly, we obtain a formula for the symmetric solution of the 
Neumann problem (13.12) on a symmetric region D, in terms of the 
symmetric harmonic kernel function. 


Theorem 13.23. Let D be a symmetric region bounded by a finite number 
of o-rectifiable Jordan curves. Let g be a symmetric, continuous function on 
OD. If u is harmonic in D and g is its inward normal derivative on OD, then 
up to an additive constant, 


(13.20) 


Proof. It is similar with the proof of the Theorem 13.22. Here we use 
Theorem 13.20. Oo 


The next theorem yields a formula for the symmetric solution of the 
Dirichlet problem (13.3) on a symmetric region D, in terms of the 
symmetric harmonic kernel function. 


Theorem 13.24. Let D be a symmetric region bounded by a finite number 
of o-rectifiable Jordan curves. Let f be a symmetric, continuous function on 
OD. There is a unique symmetric function u on D, which is harmonic on D, 
continuous on D, such thatu = f on OD. For all ¢ in D, 


(13.21) 


Proof. By Theorem 13.19, for all ¢ € D, 
OK p(z, 
u(¢) = — f u(z) edo. 


Replacing ¢ with k(¢) we get, for all ¢ € D, 


u(k es ao z) Re) do. 


Adding the last two equations and dividing by 2, it follows that 


ste) ret) 1 f u(z) oe) 4 one k(¢)) | do, 
aD 

for all ¢ € D. 
By Proposition 13.2, uis a symmetric function on D, then the left-hand 


side of the last equality is u(¢) and we conclude that for all Cin D, 


u() = —F f u(e)| PEO 4 RoE HO | ay, 
OD 


The uniqueness of the solution of the Dirichlet problem for harmonic 
functions implies (13.21). Oo 


13.4.4 Integral representations on the Klein surface 
Let ¢ be a point inside 2. A Neumann function Ng(Z, C) for the region 2, 
with singularity at ¢ is defined by 
se k k z 
Nol2,0) = Np? (2,0) = NPR), O), 
(13.22) 


where Z = 77(2). 


Remark 7. By Proposition 13.17, it follows that Np(Z, ¢) is well-defined on 
Q2. 


Therefore, N(Z,¢) is a harmonic function on Ate \ which has a 


constant normal derivative a. = on the boundary Of? and has a logarithmic 


pole at the point ¢ = x(C). 
Next, we derive the solution of the Neumann problem (13.11) on the 
region 2. 


Theorem 13.25. Let G be a continuous real-valued function on 092. Then, 
up to an additive constant, the solution of problem (13.11) 1s the function U 
defined by the relation u = U ‘1, where x is the canonical projection of O7 
on X and u is the solution (13.19) of the problem (13.12) on the symmetric 
region D, with the inward normal derivative g given by g = G’x on OD. 
Proof. The symmetry of the function u on D, yields 


AU (¢) = Au(¢) = Au(k(¢)) =0 for all€ € Q, 


where ¢ = 7(C). 
Using the symmetry of the function g on OD, we obtain 


YG) = #6 =9(0) = o(k() = GO, 


for all a € O92. Then, up to an additive constant, the function U defined on 
Q by 


U(C) = u(¢) = u(k(¢)), 
for all ¢ in Q, is the solution of problem (13.11). o 


Let ¢ be a point inside ©. The harmonic kernel function K ¢(Z, C) of the 
closed orthonormal system (y;),, for the region , with respect to the point 


¢= ie (6) } is defined by 
Ko(2,0) = Kp)(z,0) = Ky (A(z), 0), 2 = m(2) € @. 


Remark 8. By Proposition 13.21, it follows that Kq(Z, é), is well-defined 
on 02. 


The symmetric solutions on O, determine the solutions of the similar 


problems on the Klein surface X. Thus, we obtain the solution of the 
Dirichlet problem (13.2) on the region ©, with respect to the harmonic 
kernel function, for the region (2. 


Theorem 13.26. Let F be a continuous real-valued function on the border 
O92. The solution of the problem (13.2) with the boundary function F is the 


function U defined on 22, by the relation u = U’ x, where x is the canonical 
projection of O, on X and u is the solution (13.21) of the problem (13.3) on 


the symmetric region D, with the boundary function f given by f = F'n. 
Proof. By definition, 


AU(¢) = Au(¢) =0 forall€e Q, 


where ¢ = 7(C), thus U is a harmonic function. The symmetry of the 
function fon OD, implies 


U(¢) = u(¢) = F(¢) = F(A(Q)) = F() for all¢ € a2. 


Due to the uniqueness of the solution, the function U defined on Q by 


U(¢) = u(¢) = u(k(¢)), 
for all Cin 2, where ¢ = x(C), is the solution of problem (13.2) onQ. 


The next theorem gives the solution of the Neumann problem (13.11) on 
the region ©, with respect to the harmonic kernel function, for the region 92. 


Theorem 13.27. Let G be a continuous real-valued function on the border 
O92. Then, up to an additive constant, the solution of the problem (13.11) 


with the normal derivative G on 092 is the function U defined on 22, by the 
relation u = U7, where 1 is the canonical projection of O7 on X and u is 


the solution (13.20) of the problem (13.12) on the symmetric region D, with 
the normal derivative function g given by g = G’x on OD. 


Proof. By definition, 
AU(¢) = Au(¢) =0  forall€ € Q, 


where ¢ = 7(C), thus U is a harmonic function. The symmetry of the 
function g on OD, implies 


SO = BO = 9(¢) = g(k(0)) = GO, 


for all ¢ € O92. Thus, up to an additive constant, the function U defined on 
2 by 


U(¢) = u(¢) = u(k(¢)), 


is the solution of the problem (13.11) on Q. Oo 


Concluding remarks 


The methods developed in this paper remain valid in the case of all 
differential operators associated to conformal invariant metrics. Such an 
example corresponds to the invariant Laplacian (or sometimes the Laplace- 
Beltrami operator for the Poincaré-Bergman metric), see Krantz [18, 
Section 6.5]. We also refer to the pseudo-hyperbolic metric, which is 
conformally invariant, but it does not arise from integrating an infinitesimal 
metric (that is, lengths of tangent vectors at a point). A comprehensive 
analysis of the pseudo-hyperbolic metric on the disc may be found in 
Krantz [16]. 

To the best of our knowledge, there are not further results involving 
either linear or non-linear elliptic equations on Klein surfaces. This study 
can include qualitative and quantitative properties of solutions but also 
related singular or degenerate phenomena. We consider that the 
mathematical analysis of these classes of PDEs on Klein surfaces is a very 
rich and attractive research field at the interplay between complex analysis 
and non-linear analysis. 
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Riemannian manifolds, 182 
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